The identification of rater effects on open-ended math questions rated through standard rubrics and rubrics based on the SOLO taxonomy in reference to the many facet Rasch model

The purpose of the present study was to identify rater effects on open-ended math questions rated through standard rubrics and those based on the SOLO taxonomy in reference to the many facet Rasch model. The participants were 104 eight grade students and seven mathematics teachers serving as raters. The data collection instruments involved a mathematics achievement test comprised of eight open-ended questions composed by the researcher, standard rubrics, rubrics based on the SOLO taxonomy, the Survey of the Views on Standard Rubrics (SVSR), and the Survey of the Views on Rubrics based on the SOLO Taxonomy (SVRST). The data obtained from the rating of the responses to the open-ended math questions through standard rubrics and those rubrics based on the SOLO taxonomy were analyzed on the basis of the many facet Rasch model. On the other hand, the responses of the raters to the close-ended items included in the SVSR and SVRST were analyzed through arithmetic mean values whereas the responses to the open-ended items were analyzed descriptively. The findings suggested that both the standard rubrics and those based on the SOLO taxonomy could highly reliably distinguish among students of varying levels and that there was a significant difference between the items in terms of their difficulty. The rating through the standard rubrics did not suffer from the halo effect, inconsistency, rater x individual bias, or rater x individual x item bias. Nevertheless, the central tendency effect and rater x item bias were prevalent, there were significant differences among the raters in terms of severity and leniency, and the rater reliability was low. On the other hand, the rating through the rubrics based on the SOLO taxonomy were observed to be free from such rater effects as rater severity and leniency, the central tendency effect, the halo effect, inconsistency, and bias. Furthermore, the rater reliability was high, and the raters had similar levels of severity and leniency. Key words: Open-ended math questions, Rater effects, Rubrics, SOLO taxonomy, Many facet Rasch model