Evaluating state-of-the-art nmt on football tweets with high context: correlating human (MQM) and automated (COMETKIWI-XL) metrics

Adopting a correlation-centered evaluation framework, this thesis investigates how well the reference-free metric COMETKiwi-XL aligns with human MQM 2.0 judgements. Within this framework, the study compares segment- and system-level relationships between normalized MQM points (RQS) and COMETKiwi-XL scores. The sample of the study is English↔Turkish football UGC—tweets rich in slang, abbreviations, emojis, and culture-specific references. A small, domain-focused dataset of 20 tweets (10 per direction) was translated by three machine translation systems (DeepL, ChatGPT-o3, and X Translate), yielding 60 outputs. Three expert annotators applied MQM; category×severity counts were converted to APT → PWPT → RQS (0–1) and then averaged per segment. We compared these human scores with COMETKiwi-XL using Pearson's r and Kendall's τ, at both the segment and system levels. Findings suggest moderate yet inconsistent segment-level correlations, partly driven by floor effects (multiple items with RQS = 0.000) that compress variability. At the system level, MQM and COMETKiwi-XL converge to a larger extent in EN-TR but diverge substantially in TR-EN, yielding a reversed ordering of systems. Regarding MQM error tags, the distribution was balanced across translation directions. Although ChatGPT-o3 generally achieves higher RQS than the other systems, the central takeaway of this study is methodological rather than system-specific. MQM, by design, can reduce poor translations to very low scores, sometimes reaching 0.000, which indicates complete failure. COMETKiwi-XL, in contrast, distributes outputs on a smoother scale, placing scores closer together and reflecting quality differences in a more gradual manner. Given the limited size and single domain, the results should be interpreted cautiously. Even so, they provide some indications of how human-in-the-loop MQM and a reference-free automatic metric might be used together, while also pointing to the need for larger, multi-domain datasets to better examine MQM–COMET(Kiwi-XL) alignment.