Comparison of classification accuracy and parameters of cognitive diagnostic and multidimensional item response models

This study investigated the classification accuracy of person parameters estimated by 3PL multidimensional item response theory (MIRT) and cognitive diagnostic models (CDMs) via simulation and real data analysis. To compare the models, RMSE, bias, and correct classification rates of attributes and attribute profiles were calculated. As a result of the study, it was found that for all test conditions, 3PL MIRT had the lowest RMSE and bias values. The increase in test length and the correlation between attributes decreased the RMSE and test length had the greatest positive impact on decreasing RMSE values than the correlation between attributes. By increasing test length and the correlation between attributes, bias tended to decrease except for a few exceptional cases. The use of Q-matrix which was designed to measure the attributes and attributes profiles in equal number, increased the RMSE and bias for the attributes 1 and 4 which were measured by more items, and reduced RMSE and bias for other attributes. For all test conditions, 3PL MIRT had the highest attribute and attribute profile correct classification rates. It was found that the correct classification rates based on attributes ranged from 0.728 to 0.890 and the correct classification rates of attribute profiles were between 0.135 and 0.711. As the test length and correlation increased, attribute and attribute profile based correct classification rates increased monotonically. Similarly, the use of the Q-matrix which measured the attributes and the attribute profiles in equal number, increased the correct classification rates. In real data applications, G-DINA model had the best relative and absolute model fit and item fit among the other CDMs. When examining the attribute mastery probabilities, it was found that the majority of the examinees were clustered in (1,1,1) and (0,0,0) attribute classes. Keywords: cognitive diagnostic models, multidimensional item response theory, retrofitting, correct classification rate