The correspondence between test developer's presumptions, test experts' assumptions, and test takers' perceptions on reading comprehension tests

The purpose of this study was to explore the reading comprehension (RC) construct in an adult English as a Foreign language (EFL) setting. The research explored the divisibility of the RC construct into certain subskills in order to make a comprehensive assessment plan for it, taking various stakeholders into account. Building on the previous research in the assessment of reading skills, this study examines the correspondence between the test developer's, test users', and test experts' presumptions regarding reading skills, and narrates a validation process of the construct. To this end, the study theoretically defined and operationalized the RC construct by, first, devising RC tasks prior to collecting evidence for validation. Two reading comprehension tests focusing on reading sub-skills were developed, as data collection tools, for two different language ability levels. They were piloted with 60 participants and showed acceptable psychometric qualities. The main data was collected from 262 adult EFL undergraduate students at a foundation university in Istanbul, Turkey. Furthermore, for various validation procedures, more than 35 experienced teachers and 31 EFL test experts/researchers in Applied Linguistics contributed to various stages of this study. The data were analyzed quantitatively using various multivariate statistical techniques. The findings indicated that, although test experts and the test developer agreed on most subskills, no such correspondence was observed between the test developer and the examinees. It was also revealed that test takers' perceptions of reading sub-skills were related to their performance on the items in which they correctly identified the sub-skill that the test developer intended to assess. The findings support the reconceptualization of the subskill patterns underlying the RC construct at two different language ability levels. More specifically, the findings of the present study led to the establishment of nine reading subskills at two different language proficiency levels. Also, the findings help to confirm the divisibility of the construct of reading into interrelated and overlapping sub-skills, rather than independent ones. In addition, existing reading subskills taxonomies were refined and this could be a significant contribution to the field, since the outcome will provide language test developers with a means to ensure their tests' construct representation. Among the pedagogical implications of these findings is that sub-skill training of EFL learners will result in improved achievement in reading skill. This need should be reflected not only in classrooms but also in the curriculum and instructional materials, as well.