Comparing scores, quality, and difficulty of AI-generated and human-designed reading comprehension tests for EFL students

This study investigates the psychometric properties, reliability, validity, and item-level quality of reading comprehension tests generated by artificial intelligence (AI) models compared to a human-designed TOEFL reading test. Across five separate sessions, student groups completed one of four different test types: a human-designed TOEFL test or AI-generated tests created by ChatGPT, Claude, and Perplexity. A mixed-methods research design was employed, combining quantitative analysis, including Mann-Whitney U tests, item facility (IF) and item discrimination (ID) calculations with qualitative expert evaluations assessing subskill alignment, content validity, and item-writing guideline adherence. Results demonstrated that the human-designed TOEFL test consistently exhibited superior psychometric performance, with higher reliability, balanced difficulty, and stronger discrimination indices across items. In contrast, AI-generated tests showed greater variability, with some items displaying acceptable metrics while others revealed deficiencies such as distractor implausibility, stem ambiguity, and subskill misalignment. These findings align with recent literature indicating that while AI models can generate assessment items with surface-level validity, they often lack deeper cognitive coherence and psychometric robustness. The study explores the effectiveness and quality of AI-generated multiple-choice reading comprehension tests by comparing them with human-designed versions. It highlights both the potential and current limitations of AI-generated items, offering insights for improving testing practices in high-stakes reading comprehension assessments.