Rasch analysis of teacher, artificial intelligence and hybrid feedback on EFL students' writing

Recently there has been a remarkable rise in interest regarding artificial intelligence (AI) tools in educational settings, particularly within language teaching and learning environments. As these technologies evolve, their potential to enhance traditional feedback mechanisms, particularly in writing instruction, is being explored extensively. This study investigates the effectiveness of types of writing feedback—teacher, AI and hybrid feedback—on tertiary-level English as a foreign language (EFL) students' writing. Specifically, the research focuses on the feedback categories including higher- and lower-order concerns in EFL instruction. The study employed an explanatory mixed-methods approach, integrating quantitative analysis of feedback types and qualitative insights from semi-structured interviews with EFL instructors. 80 essays written by tertiary-level EFL students were analyzed, with feedback from four experienced EFL teachers and an AI-based tool. A rubric adapted from existing literature categorized and assessed the feedback units. The results were statistically analyzed, incorporating Rasch analysis, to identify the interaction among feedback types and feedback categories. The results suggest that AI effectively provides feedback on lower-order concerns (LOCs) while less effective when addressing higher-order concerns (HOCs). On the other hand, the teachers effectively provide feedback on HOCs at the discourse level and they are less effective when addressing LOCs due to the workload of grammar, vocabulary and mechanics. Therefore, the hybrid feedback type, which involves EFL teachers supplementing AI's feedback, demonstrated potential for improving the HOCs of students' writing, as well. The qualitative phase of the study was conducted by semi-structured interviews with EFL instructors to discover their perspectives on using AI-generated feedback following the completion of the quantitative analysis. This study demonstrated AI's capability to help reduce the workload of EFL teachers when giving feedback on students' written work at the sentence level. Thus, EFL teachers can concentrate more on the HOCs of EFL students' writing. This study concluded that AI-driven feedback integration with traditional teacher feedback could result in more effective writing instruction in EFL contexts. Contributing to the growing interest in AI in education, the research provides valuable insights for EFL teachers on how to incorporate AI technologies into their writing lessons.