The investigation of items measuring high-level thinking skills in terms of student score and score reliability

The aim of this study is to evaluate the answers given by the 8th grade students to the open-ended and multiple-choice items determined according to the 5th, 6th, 7th and 8th grade mathematics course achievements, which measure high-level thinking skills according to Bloom's taxonomy, by four raters according to the scoring keys. is the examination of reliability between raters. For this purpose, PISA questions applied in 2006 and 2012 were used. The suitability of the selected PISA questions for the Bloom taxonomy levels and the 5th, 6th, 7th and 8th grade mathematics course outcomes was determined by the researcher, and expert opinion was taken for their suitability. After receiving expert opinions, open-ended and multiple-choice questions with the same question root were applied to 8th grade secondary school students in the Polatlı district of Ankara in the 2020-2021 academic year at certain time intervals. Analytical rubric and holistic rubric prepared by the researcher were given to raters to use during scoring. Reliability estimations were made according to Classical Test Theory and Generalizability Theory of the scores obtained as a result of scoring the raters using rubrics. The items answered by the students were scored by four volunteer mathematics teachers according to analytical and holistic scoring keys. In the analysis of the data, while the reliability was determined according to the Classical Test Theory, the Pearson correlation and Cronbach Alpha (α) coefficients were calculated with SPSS 25. According to the Generalizability Theory, the G coefficient and the percentage of total variance explanation were calculated with the Edu-G 6.1 program. As a result of the analysis, the correlation between the answers given by the students to the open-ended and multiple-choice items was found at a low level. According to Classical Test Theory and Generalizability Theory, the consistency between the scores obtained as a result of scoring open-ended items according to the analytical rubric and the scores obtained as a result of scoring them according to the holistic rubric was found to be at a high level.