Designing a multistage test for assessment literacy

Assessment literacy is a critical component of effective teaching and learning, as it underpins the validity, reliability, and fairness of educational assessments. Measuring the assessment literacy of in-service and pre-service teachers is essential for understanding their knowledge, skills, and competencies in implementing sound assessment practices. However, existing theoretical frameworks and assessment tools often fail to capture the multifaceted nature of this construct. This study aims to bridge this gap by extending a widely used assessment literacy framework into an assessment framework and developing a multistage test for measuring assessment literacy efficiently. In the first phase, insights from 19 expert interviews and an extensive literature review were used to construct an 8-dimensional assessment framework for assessment literacy. The extended framework incorporated two novel dimensions: competency in technology-based assessment and competency in developing assessment tools. A 105-item pool was then developed based on this framework and administered via paper-and-pencil format to 1,223 pre-service teachers from 11 universities to estimate item parameters. A 1-3-3 panel multistage test was developed on the Concerto platform, based on Item Response Theory (IRT) analysis of pilot test findings. Each module in the MST included one item per dimension and met targeted psychometric criteria. The MST was subsequently administered to 106 pre-service teachers. IRT-based analysis confirmed the effectiveness of the MST in measuring assessment literacy across a range of ability levels, with a decreasing standard error across stages, demonstrating both efficiency and precision.