Automatic scoring systems: A systematic review

Requirements such as speed, consistency and objectivity in measurement and assessment processes in education are increasing the importance of automated text assessment systems from day to day. Against this background, this study aims to analyse the current status of automated text scoring systems and their contribution to education by means of a systematic literature review using text mining methods. In the study, 397 articles retrieved with specific keywords from academic databases such as ACM, ACL, IEEE Explore, Springer and Science Direct were reduced to 182 articles according to the previously defined inclusion and exclusion criteria. These articles were analysed in detail according to categories such as datasets used, automatic scoring types, evaluation metrics, content and machine learning techniques and classified in a comprehensive data table. In addition, text mining techniques based on latent Dirichlet Mapping were used to analyse the most important topics and trends in the articles and visualise them with word clouds. As a result of the systematic literature research, the most important trends and focal points in automatic evaluation systems were identified in detail. When analysing the publication years of the studies, it was found that the number of studies conducted in this area has increased significantly, especially since 2010. When analysing the data sets, it was found that large and diverse sources such as TOEFL, ASAP and Kaggle were used intensively in the studies. Different types of automated assessment were analysed and it was found that "semantic"," "deep learning" and "rubric-based" approaches were predominant. It was also found that modern methods such as "neural networks"," "BERT" and "LSTM" were often favoured among the machine learning methods. In terms of verification metrics, it was found that reliability measures such as "Quadratic Weighted Kappa (QWK)" and "Correlation" were widely used. Considering the limited number of academic studies conducted in this field in Turkey, this study serves as a guide for researchers and decision makers in the field of education for the use of these technology-based applications in the field of measurement and assessment in education as well as for the literature.