The impact of calibration on scoring reliability and rater severity in a business English written exam

This study investigates the impact of calibration on inter-rater reliability, intra-rater reliability, and rater severity in an online written examination. Six raters, with backgrounds in English Language Teaching, marked the same set of 32 exams in two phases, with a calibration meeting held in-between. Intraclass Correlation Coefficient analysis, T-tests, Many Facet Rasch Measurement, and descriptive statistics were used to evaluate the impact of calibration on reliability and severity. The findings suggested that calibration significantly enhances inter-rater reliability and scoring consistency while also highlighting the complexities of intra-rater reliability when calibration is introduced. Notable shifts in rater severity levels were also observed post-calibration. These findings have implications for ensuring reliability and consistency in evaluation processes. The results underscore the potential of calibration in improving the scoring reliability and reducing rater severity effects, contributing to a more reliable assessment process in corporate and educational settings.