Rater discrepancy resolution in second language writing assessment: An analysis of rater negotiations

The purpose of this study was to analyze rater negotiations as a way to resolve score discrepancies in writing assessment with a view to understand the nature and usefulness of the method. For that purpose, rater negotiations following a B2 level module exit exam at the language school of an English-medium university in Turkey were analyzed. The data came from 30 Turkish EFL teachers and it was transcribed, analyzed and categorized into discussion moves. The Rater Negotiation Scheme (RNS) was developed through a recursive analysis and categorization process, and it was validated through field testing in authentic settings. The validation process of the RNS consisted of two phases: a) training of the participants for using the RNS to code sample transcribed rater negotiations, and b) asking the participants to code five purposefully sampled rater dialogues by using the RNS. At the end of the field testing, the RNS was validated with 89% participant coding-key agreement. In addition to the main purpose of this study, the following issues were also addressed. First, through the use of the rater moves listed in the RNS, the study investigated the stages of rater negotiations and the rater moves within them to further identify the dynamics of the process. Three functional parts of rater negotiations namely the opening stage, the discussion stage, and the concluding stage were identified, and the types of moves used at each stage were listed. Secondly, the rater negotiations were analyzed to investigate how rater tendencies towards accuracy vs. fluency orientation affect the scoring decisions of rater pairs. The results showed that a majority of the raters were fluency oriented in the examined cases. However, for the cases where linguistic features of the test taker essay frequently reported to be problematic, an overriding effect of accuracy was observed, and the winners of these cases were always the accuracy-oriented raters. Third, the study also investigated the raters' reports on factors that cause rater discrepancy, and presented a list of fifteen discrepancy creating factors. The results obtained throughout this study had significant implications for the contexts in which rater negotiations are used for score discrepancy resolution especially in terms of rater training, writing task development and scale use.