A comparative analysis of AI-tools and English teachers in writing assessment

This research focuses on comparing the evaluation of writing tasks by artificial intelligence technologies and English language teachers. The increasing role of advancing technology in all areas of education and the recent surge in the use of AI tools have been the primary motivations for this study. The research aims to thoroughly examine the similarities and differences between the evaluations conducted by AI and those by English language teachers, providing insights into the possible reasons behind these outcomes. Additionally, this research also examined the differences between the AI enhanced chatbots that are ChatGPT, BingAI, and Gemini, in terms of grading and giving feedback. The study uses a mixed-method research design considering grading as quantitative and feedback as qualitative aspect of the data. During the data collection process, ten first-year teacher candidates from Bolu Abant Izzet Baysal University were asked to write three different types of essays which are narrative essays, expository essays, and argumentative essays. These essays were then evaluated by five English language teachers and three AI tools that are ChatGPT, BingAI, and Gemini. This process of evaluation will be conducted using TOEFL IBT Writing evaluation rubric considering four main criteria which are Task Response, Organization, Coherence, and Language Use. In addition to the four criteria, to get more elaborate and comprehensive grading and feedback "Overall" part will be added as another criterion. A comparative analysis of the results was conducted, and the underlying causes are identified. The results indicate that in terms of grading and giving quality feedback, chatbots performed on-par with teachers despite some noticeable differences. The findings of this research offer valuable insights into the advantages and limitations of AI technologies. Furthermore, it will provide a meaningful perspective on how these technologies can be effectively integrated into educational programs.