Investigation of interrater agreement and eye tracking in performance evaluation

In this study, the fit between raters, coefficients of fit, generalizability and many facet Rasch were investigated in the performance evaluation process. For this purpose, to evaluate the presentation performance of the students, the relationships between the scores given by the same raters in the classroom during the performance and the scores given on the screen by eye tracking technique were examined. The scores obtained as a result of the in-class evaluation and the evaluation on the screen with the eye tracking technique were analyzed according to the inter-rater fit statistics in the classical test theory, the generalizability theory and the many facet Rasch model. It was found that there was a good agreement between the raters in the presentations of idealism and realism. According to the results of the generalizability analysis, the biggest source of variance was the presentation. As a result of the many facet Rasch model, it was found that the scores were consistent with each other and the scores differed significantly according to the presentation and the items, but did not differ significantly according to the application. The mean fixation times of eye tracking metrics were examined by many facet Rasch model analysis and it was found that the raters did not show a significant difference according to the mean fixation times. As a result, it was found that generalizability and multi-surface Rasch model results were compatible with each other. The study indicated that it could be used as an alternative method to investigate the compatibility between eye tracking metrics and raters. Keywords: eye tracking, inter-rater agreement, generalizability theory, many facet rasch model, inter-rater reliability, performance assessment