Examination of possible bias in mathematical literacy test scores with differential item functioning and measurement invariance

Within the scope of this research, potential biases in the measurement of mathematical literacy skills were examined through DIF and Measurement Invariance analyses. Throughout the study, six core research questions were addressed, focusing on whether measurement invariance was achieved and whether the items exhibited DIF across gender subgroups (female and male students), residential area subgroups (students from metropolitan and urban regions), and school type subgroups (private and state schools) for item scores from a test measuring mathematical literacy skills. In this study, which utilized a comprehensive methodology, a study group consisting of 7179 middle school students (from 5th, 6th, 7th, and 8th-grades) who participated in the Mathematics subtest of the Matbeg exam administered in various provinces of Turkey in 2024 was examined. As part of the item analysis, Item Difficulty Indices and Item Discrimination Indices were calculated. DIF analyses were conducted using the difR package in R software, employing the Mantel-Haenszel Technique based on Classical Test Theory (CTT) and Lord's Chi-Square Statistic Technique based on Item Response Theory (IRT). The examination of measurement invariance across gender, residential area, and school type subgroups was conducted through a systematic, stepwise procedure which entailed the use of Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA) to establish the factorial structure, followed by Multigroup Confirmatory Factor Analysis (MGCFA) to assess measurement invariance, contingent upon the satisfaction of the fundamental assumptions necessary for such analyses. The findings revealed that full measurement invariance is achieved across gender groups for all grades (5th, 6th, 7th, and 8th). DIF results from both techniques revealed no gender-related bias. As for the residential area variable, full measurement invariance is achieved for the 5th grade, while structural invariance is established for the 6th grade, with DIF analyses indicating a general favor towards students from urban areas in both grades. Regarding school type, full measurement invariance is achieved for the 5th grade, structural invariance for the 6th grade, and weak factorial invariance for the 7th grade. DIF results show a balanced distribution of bias, with no notable advantage for either private or state school students. The findings suggest that potential biases favor urban and public school students in certain grades. The comparison of DIF results across multiple techniques highlights the necessity of employing diverse methods to comprehensively identify item bias. Based on these findings, factors potentially influencing item bias are identified, and recommendations are provided for future research.