An application for detecting content produced by artificial intelligence tools in academic field

The rapid development of artificial intelligence technologies, particularly in the field of natural language processing, has led to fundamental changes in text production and content editing processes. The widespread use of generative language models, especially in academic writing processes, has made it difficult to distinguish between human-generated texts and AI-supported content, raising various issues in terms of academic ethics, content security, and publishing principles. The main objective of this study is to develop a classification model that can accurately detect AI-generated content in academic texts. For this study, abstracts of Turkish academic articles published before 2022 were selected. A unique dataset was created using 2,000 Turkish academic articles published in journals indexed in databases such as ULAKBİM, Scopus, and Web of Science, covering various fields such as health, engineering, educational sciences, and social sciences. In addition to the original texts, the dataset was expanded to include three classes (Real Data, Artificial Intelligence, and Edited): summaries generated using generative artificial intelligence tools such as ChatGPT, Smodin, and Gemini, and rephrasings of the original summaries by generative artificial intelligence applications. This resulted in a balanced dataset of 6,000 items. For the classification process, deep neural network-based models widely used in the literature, such as XLM-Roberta, TurkishBERT, and Bert Base Multilingual, were employed. The performance of the models was evaluated based on criteria such as accuracy, precision, sensitivity, and F1 score. The models' classification performance was compared by analyzing them according to these metrics. When the results were evaluated, the XLM-Roberta model achieved the highest accuracy value of 97.22%, the Bert Base Multilingual model achieved 96.40%, and the TurkishBERT model achieved 90.22%. The findings show that the developed classifier is effective in detecting artificial intelligence content in academic texts. The high performance provided by the models used can be considered an important step in both checking the originality of academic content and increasing original scientific productivity. This study contributes to academic ethics and helps to increase transparency and reliability in scientific studies.