Analysis of interrater reliability with different methods according to rater experience

The aim of this study was to compare the reliability values obtained from the scoring of open-ended items measuring writing skills by teachers with and without experience in the scoring process with the use of analytical and holistic rubrics. Within the scope of this research, in which it is tried to determine whether there is a difference between experienced and inexperienced groups in the levels of reliability and consistency determined by different techniques, the "Writing Skills Test" data conducted by the General Directorate of Measurement, Evaluation and Examination Services of the Ministry of National Education were used. For this purpose, the results obtained from the writing test which is applied to a total of 240 students studying in the 4th, 7th and 9th grades were scored by two separate experienced an inexperienced groups. Reliability estimations of the scoring results based on Kappa technique, Krippendorff's Alpha coefficient and Generalizability Theory were made and the level of consistency between raters was investigated. As a result of the research, parallel and consistent values were observed in the coefficients obtained from Kappa and Krippendorff's alpha technique in all groups. When the G and Phi coefficients obtained from the generalizability theory studies were examined, it was observed that the expected (0,80) value for reliability and generalizability was reached in inexperienced Turkish teachers and both groups (experienced and inexperienced) of Turkish language and literature teachers. However, there was no evidence of a significant effect of scoring experience alone on reliability.