The use of artificial intelligence in l2 reading test development: A study of language instructors' perceptions and experiences in an English preparatory school

This exploratory case study aimed to investigate English language instructors' experiences and perceptions of using ChatGPT-4 for L2 reading test development. To this end, four language instructors working in an English preparatory school of a state university in Türkiye, with testing experience, participated in the study. They were asked to use ChatGPT-4 to prepare an L2 reading test, including two reading tasks with five multiple-choice questions: one at the elementary level and the other at the intermediate level. Data were collected through screen recording of instructors' test development process, think-aloud protocols, and ChatGPT conversation transcripts. Data were analyzed using a content analysis approach. The findings demonstrated some variation among the instructors in how they engaged with ChatGPT-4 for L2 reading test development, despite the small number of participants. A large proportion of all instructors' prompts dealt with item quality and distractor quality, rather than text quality, where they tried to fix more item and distractor issues in the intermediate-level reading task than in the elementary-level task. The way the instructors carefully checked the ChatGPT output for several aspects made it obvious that human agency and expertise have a crucial role in ensuring that the AI-generated test materials meet the required standards. While their perceptions were mostly positive about the reading texts that ChatGPT generated, except for a few length issues, their comments during test development with ChatGPT-4 revealed negative perceptions regarding several aspects. The instructors perceived their intermediate- level reading task development experience more negatively than their elementary-level reading task development experience.