A comparison of classification accuracy of different text mining algorithms in classification of foreign language proficiency exam items

The aim of this study is to compare the classification accuracy of different text mining algorithms in the classification of foreign language proficiency examination items. In the study, previous test items from ÜDS English tests (N=42) held between 2006 and 2012 by ÖSYM were analysed. Natural and Applied Sciences (n=956), Health Sciences (n=956) and Social Sciences (n=956) test items (N=2868) were classified with k-Nearest Neighbours (kNN), Naïve Bayes (NB), Naïve Bayes-Kernel (NB-K), Random Forest (RF) and Support Vector Machines (SVM) algorithms. After the pre-processing step, domain specific and general word frequencies were obtained. Following the parameter optimizations, binomial and multiclass classifications of the items from three domains were executed. The binary classification accuracies were obtained as Natural and Applied Sciences vs. Health Sciences items, NB-K (90.60%), kNN (90.23%), RF (86.84%), SVM (86.84%) and NB (83.08%); Natural and Applied Sciences vs. Social Sciences items, kNN (89.10%), NB-K (88.35%), SVM (85.34%), RF (84.59%) and NB (78.20%); Health Sciences vs. Social Sciences items, SVM (92.48%), kNN (91.35%), RF (%89.10%), NB (87.22%) and NB-K (86.47%). The multiclass classification accuracy was obtained as kNN (84.96%), NB-K (83.96%), RF (80.45%), SVM (78.70%) and NB (71.93%). Binomial and multiclass classifications were repeated to determine the effects of the removal of the items, which were determined to be damaging the classification accuracy by expert opinion and cross-validation method, from the data set. Significant increase in classification accuracy in the binomial classifications, NB (X̄=9.83%), NB-K (X̄=9.12%), RF (X̄=8.82%), SVM (X̄=8.75%) and kNN (X̄=7.39%); in the multiclass classification, NB (17.89%), SVM (15.31%), NB-K (12.15%), RF (11.47%) and kNN (10.58%) was obtained (p<.05). kNN was determined to be the least affected from the items that damage the classification accuracy.