A study on the evaluation and the development of computerized essay scoring rubrics in terms of reliability and validity

This study investigated the validity and the reliability of essay scoring rubrics used for intermediate and upper-intermediate levels at Zirve University English preparatory school. The rubrics were examined in terms of content validity and inter-rater reliability. In order to determine the content validity of the rubrics, a focus group interview was held with the participation of five writing skill instructors as the first research question. The aim was to what extent the writing class objectives and the descriptors of essay scoring rubrics matched each other. The focus group concluded that the rubrics were compatible with the writing class objectives of the preparatory school, but the descriptors of the rubrics needed to be re-designed. For the second research question, totally 351 C (intermediate) level and D (upper- intermediate) level students' essays were analyzed by using Pearson r correlation coefficient in order to see the inter-rater reliability between graders of the rubrics. The analysis results showed that the correlation between graders was low as Pearson r results for total scores of the students? essays were r= .623 for C level and r= .552 for D level at the 0.01 level. As the third research question, one more focus group interview was held with the same participants in order to develop a new institutional essay scoring rubric for Zirve University English preparatory school. The focus group developed a new essay-scoring rubric by taking the results of the first two research questions into consideration. The newly developed rubric was also analyzed by Pearson r correlation coefficient in order to see the inter-rater reliability between graders. 59 C (intermediate) level and D (upper-intermediate) level students' essays were used for that analysis. The analysis results showed that the correlation between graders was higher than the present rubrics as Pearson r results for total scores of the students? essays were r= .848 at the 0.01 level. As a result, the newly developed essay-scoring rubric provided more reliable results than the present rubrics used at the preparatory school. It may be concluded that having an institutional rubric, which is compatible with the needs and the objectives of the institution, meets the expectations and provides more consistent grading results.