Implementation and evaluation of Chatbots based on artificial intelligence in science education

The purpose of this study is to examine the impact of the AI-supported Astrosciencebot application, developed for use in the education of science teacher candidates, on the learning process and its role in the process, examining its effect on academic achievement, knowledge retention, elimination of misconceptions, 21st-century skills, perceived usability, behavioral intentions, and user opinions. The study adopted mixed-methods research design. A case study design was used for the qualitative dimension, while a pre-test-post-test control group quasi-experimental design was used for the quantitative dimension. The study was conducted over a total of 16 weeks with 32 teacher candidates (EG: N=16, CG: N=16) enrolled in the Science Education Department of the Faculty of Education at Ordu University during the fall and spring semesters of the 2024-2025 academic year, selected through purposive sampling. During the data collection process, the three-stage KTT, the Multidimensional 21st Century Skills Scale, the Chatbot Usability Scale, the Chatbot Usage Behavioral Intent Scale, the Semi-Structured Interview Form, and the Process Evaluation Form were used. In the analysis of the data, descriptive and inferential statistical methods, normality tests, Dependent Sample T Test, and Independent Sample T Test were used for quantitative data, while content analysis was applied for qualitative data. According to the quantitative findings of the study, the learning success, misconceptions, 21st-century skills, usability perception, and behavioral intentions toward chatbot use of teacher candidates in the experimental group using AstroScienceBot were positively affected. In the experimental group, learning levels increased and misconceptions decreased compared to the control group, and these differences persisted at the end of the tests. Significant improvements were observed in 21st-century skills, and the chatbot's accessibility, usability, and reliability were found to be positive, while the quality of conversation and presentation of information provided moderate satisfaction. Behavioral intentions toward chatbot use were high, with strong opinions on factors such as performance expectations, attitudes, and social influence. Qualitative findings showed that teacher candidates found the chatbot reliable and easy to use, with advantages such as 24/7 access, instant feedback, and effectiveness in correcting misconceptions. Chatbots have demonstrated their functionality in science education by increasing interest and motivation and supporting teaching materials. However, some participants reported negative views in terms of permanent learning and thinking skills, and trust in artificial intelligence was sometimes questioned. In conclusion, the AI-supported chatbot application has been effective in improving teacher candidates' learning success and reducing misconceptions in science education while developing 21st-century skills. Its usability was found to be satisfactory, and it has fostered positive attitudes toward the use of chatbots in education.