Applying automatic item generation in family medicine field and evaluating generated test items

Multiple-choice items form the basis of many assessments used in medical education. Writing questions using traditional approaches is, however, a time-consuming and challenging process. Therefore, the primary aim of this study is to generate case-based multiple-choice items related to preventive medicine using the artificial intelligence model ChatGPT-4 and to examine the psychometric properties of the generated items. Of the 25 items produced by ChatGPT-4, 20 were removed from the study after being reviewed by field experts, as they did not meet the required characteristics of a high-quality multiple-choice item. The remaining five questions were administered to 110 family medicine residency students, and item statistics were obtained based on classical test theory. Evaluations by field experts revealed that while the stems and options of the generated items adhered to high-quality item writing standards, the distractors needed improvement. Item statistics based on student responses indicated that the first item was too easy and not discriminatory, while one of the remaining four items was easy and the other three had moderate difficulty and were discriminatory. Distractor analyses showed that for the item answered correctly by 97.3% of students, none of the distractors were effective, whereas for the other four items, one or two distractors were marked by less than 5% of the students. In conclusion, ChatGPT can assist field experts in creating case-based multiple-choice items for medical education; however, it is essential that the generated items are reviewed by experts before use.