Determination of inter-rater reliability of secondary education English proficiency using generalization theory and Rasch Model

In this study; As a result of scoring the English proficiency scores of 12th grade secondary school students in the 2022-2023 academic year by different raters with and without a holistic scoring key, it was aimed to determine their reliability and consistency with the "Generalizability Theory" and "Multivariability-Based Rasch Model". In Generalizability Theory, the validity and comparison of decision studies were examined by considering cross-over design. The data was handled with the multivariability model through the Rasch model. While using these theories, school, gender and scaled and non-scaled evaluation variables were taken into account and the analyzes were carried out entirely on real data. In order to evaluate the English proficiency that is the subject of the research, it was applied to 124 students in the 12th grade in 4 different secondary schools, which were planned to be evaluated by 3 different experts in order to score the composition writing skill at the 12th grade level. The answers given by these students were scored by 3 different raters, one with a scale and the other two without a scale. In the next stage of the research, the reliability of the scores obtained was examined according to "Generalizability Theory" and "Multivariability Rasch Model Theory". In Generalizability Theory, reliability coefficients were calculated using the bxp (b: individual, p: rater) crossed design. Separate reliability calculations were made for individual, rater and school dimensions with the Multivariability Rasch Model. EduG 6.1 (Swiss Society for Research IN Education Working Group, 2010) computer program was used for the calculation process, and FACETS (Linacre, 2007) program was used for the MLRM analysis. According to the results obtained after this research, according to the Generalizability Theory, the G and Phi coefficients of English proficiency were seen to be relatively different compared to the real situation in alternative decision studies. It was concluded that there was a significant relationship between the scores given by the raters, showing varying values in determining inter-rater agreement. According to the Generalizability Theory, the G coefficient of the first rater of the scores obtained by measuring English proficiency is 0.85, while the second and third rater's are 0.88. The Phi coefficient of the first rater was calculated as 0.79, the Phi coefficient of the second rater was calculated as 0.72, and the Phi coefficient of the third rater was calculated as 0.82. According to these results, it was observed that there were differences between raters as a result of measuring English proficiency applied in 2022. In the study conducted to measure English proficiency, it was determined that although there was a difference between the average scores of the three raters, they scored compatible with each other. Additionally, it was concluded that increasing the number of items would be more beneficial in terms of labor time and cost than increasing the number of raters. In this research, it was seen that among the theories used to measure English proficiency, two theories gave similar results. In this research, it was seen that among the theories used to measure English proficiency, two theories gave similar results. In addition, it was thought that it would contribute to the literature in order to see the difference made by the use of the scale after the raters scored with and without a scale. Key words: Generalizability theory, Rasch model, Inter-rater reliability, Holistic rubric, Performance-based status determination.