Evaluation of turkish text-based open-ended questions with artificial intelligence

In this study, an artificial intelligence-based system was developed to automatically evaluate Turkish text-based open-ended questions. In the system architecture, a six-layer model using Bi-LSTM (Bi-directional long-short term memory) was created in order to better understand the context of text data and reveal language relationships more accurately. In the system, four different text embedding methods were used to create embedded representations of the answers (BERT, S-BERT, Glove and Word2Vec). The main purpose of the study is to compare the performances of different embedding methods and to evaluate how consistent the developed system is with the ratings made by instructors. The performance of the embedding methods used in the study was measured using the Weighted Kappa Coefficient (QWK). The results showed that contextual meaning extraction methods such as S-BERT and BERT showed higher success compared to classical embedding methods such as Glove and Word2Vec. These findings revealed the importance of contextual information extraction in text evaluation. It also validated the ability of Bi-LSTM to handle such contexts. The performance of the system was compared with real instructor scores on the all student group and students at different education levels (associate and undergraduate). As a result of the analysis, it has been shown that the system gives scores close to the real instructor scores for the students in general and for undergraduate students. This finding shows that the system can be a more effective assessment tool, especially for undergraduate students. In the evaluation of associate degree students, a significant difference was observed between the scores given by the system and the actual instructor scores. This finding suggests that the system tends to score the responses of students with lower education levels lower than those of real instructors. In general, the findings of this study reveal that artificial intelligence-based automatic evaluation systems can produce results close to human evaluation. However, the fact that the system's highest QWK score is 0.68 also reveals the need for performance improvement. The number of studies on this subject in our country is very few. In this respect, the study has taken an important step in the automatic evaluation of open-ended questions in our country. Future studies in different disciplines using larger-scale data sets and more up-to-date embedding methods may further increase the generalizability and reliability of the system.