Comparison of classification performances of non-parametric cognitive diagnosis, artificial neural network and DINO Model

It is important to choose the methods that will best serve the purpose of measurement and evaluation activities. In the literature, it is seen that cognitive diagnosis models (CDMs) can give biased results about students in some conditions where there is a small sample, few items or many attributes. In order to avoid these biased results, there is a need for studies aiming to compare the classification rates of alternative methods used with each other under different conditions and to determine which method performs better under which conditions. Based on this idea, in this study, it is aimed to compare the attribute (ACR) and pattern-level (PCR) classification rates of Nonparametric Cognitive Diagnosis (NPCD) and Artificial Neural Networks (ANNs) on data sets of Deterministic-Input, Noisy-Or Gate (DINO) model-based simulation firstly with each other and then with DINO model under the conditions of the attribute number, sample size, item number and missing data rate. In addition, it is aimed to examine the similarities of the classification rates of NPCD, ANN and DINO model on the PISA 2015 collaborative problem-solving (CPS) data sets under the conditions of the attribute number and sample size. In this study, in which simulation and descriptive research design was used, simulation data sets were produced based on the complex Q matrix structure and the DINO model from compensatory models. PISA 2015 CPS competency was chosen for the real data sets and the conditions of the sample size factor were determined by simple random method among 43 countries and 18170 students participating in the PISA 2015 CPS application. In the research, firstly, the ACRs and PCRs of NPCD, ANN and DINO model on the simulation data sets were examined separately under the conditions of the present research. As a result of the analysis, it is seen that as the number of attributes (3, 5 and 7) increases, the ACRs and PCRs of NPCD and DINO model decrease whereas the ACRs of ANN increase and the PCRs of ANN decrease and then increase. When the sample size is increased (30, 100 and 500), there is no systematicity regarding the changes of the ACRs and PCRs of NPCD and ANN whereas the ACRs and PCRs of DINO model increase as the sample size increases. As the number of items is increased (15, 30 and 45), the ACRs and PCRs of NPCD and DINO model increase while the ACRs and PCRs of ANN decrease. As the missing data rate is increased (0; 0,5 and 0,10), it is observed that the ACRs and PCRs of ANN do not change in some conditions, but decrease slightly in some conditions whereas the ACRs and PCRs of NPCD and DINO model decrease in all conditions. In the next stage of the research, how the ACRs and PCRs of NPCD, ANN and DINO model change together in these research conditions is examined. As a result of the factorial ANOVA, it is revealed that NPCD has slightly lower but comparable classification rates than DINO models in all conditions while ANN always has lower rates than NPCD and DINO models. Finally, in the study, the similarity of the attribute (SACR) and pattern-level classification rates (SPCR) of NPCD, ANN and DINO model obtained from the PISA 2015 CPS data sets is examined under the conditions of the attribute number and sample size. As a result of the analysis, it is determined that as the number of attributes (3, 7 and 11) increases, the SACRs of NPCD, ANN and DINO model generally decrease and then increase whereas the SPCRs generally decrease. As the sample size increases (30, 100 and 500), no systematicity is encountered regarding the increase, decrease or stay constant of the SACRs and SPCRs of NPCD, ANN and DINO model. As a result, the similarity between NPCD and DINO model classification rates is observed in both simulation and real data sets. In addition, regarding to the increase in sample size in both simulation and real data sets, no systematicity is found in the increase or decrease of the classification rates of NPCD and ANN and the similarities of these rates. In this study, since NPCD and ANN has been recently started to be used in cognitive diagnosis, generalizable results cannot be reached regarding the changes in the classification rates of NPCD and ANN under various conditions. For this reason, it is recommended to examine how the ACRs and PCRs of NPCD and ANN change in various simulation and real data sets under various conditions.