The comparison of different techniques used to determine differential item functioning in terms of type 1 error and statistical power rates

In this study, it was aimed to compare type 1 error and statistical power rates under the conditions specified by DIF determination techniques based on CTT and IRT in items graded by dichotomous scored. For this purpose, a simulation study was conducted by using Mantel-Haenszel (MH), Logistic Regression (LD), Lord's Chi- Square, and Raju's Areas Measures. In the simulation design used in basic research model, two- parameter item response model, focus and reference group's ability distribution, and DIF type were the fixed factors while sample size (1800, 3000), rates of sample size (0.50, 1), test length (20, 80) and item rate including DIF (0, 0.05, 0.10) were manipulated factors. The data were produced for Type 1 error and statistical power rates separately in R program. The parameters a (x̄= 0.8, SS= 0.4) and b (-2, 2) in focus and reference groups were drawn from the same distribution. In order to determine Type 1 error rates by using DIF determination techniques under specific conditions, the data were produced by determining samples size as N1= 1800 and N2= 3000, the rates of sample size as R/O1= 0.50 and R/O2= 1, test length as TU1= 20 and TU2= 80. For the purpose of specifying statistical power rates, in addition to the same factors determined for Type 1 error, 0.50 was added to parameter b of certain items in the focus group for items including DIF rates. The items including DIF were produced by manipulating b parameter of reference group which was taken from focus group. As a result, data were produced showing uniform DIF. In DIF analyses, 25 replications were conducted for a total of 24 data set (2x2x2x3) produced according to two sample sizes, two rates of sample sizes, two test lengths, and three rates of items including DIF mentioned above. The analysis of the data was carried out by using "difR" pack in R program. In the analyses conducted for techniques based on IRT, predictions were made in accordance with 2PL item response design by using "ltm" command. In the current study, it was found that all techniques used for Type 1 error showed Type 1 error rates inflated above nominal alpha level. It was also demonstrated that MH had the highest error rate while Raju's Areas Measures had the lowest error rate. On the other hand, MH technique produced the highest statistical power rates. The analysis of the findings of Type 1 error and statistical power rates illustrated that techniques based on both of the theories performed better in 1800 sample size. Furthermore, the increase in the sample size affected techniques based on CTT rather than IRT. Also, the findings demonstrated that the techniques' Type 1 error rates were lower while their statistical power rates were higher under the conditions where test length was 80-item long and the sample sizes were not equal.