Comparison and evaluation of large language models (LLMS) in mathematics education

The use of large language models (LLMs), which are among the generative artificial intelligence (GenAI) systems rapidly increasing in the education sector, has become critical and important. It has become vitally significant to evaluate and compare LLMs in the context of mathematics education. This study comprehensively investigates the mathematical problem-solving capabilities of four distinct LLMs, with a focus on both final answer accuracy and step-by-step reasoning patterns. Using the Grade School Mathematics (GSM8K), a widely used dataset in mathematics education, as the benchmark, we compare the performance of multiple instruction-tuned LLMs (Gemini 2.0 Flash, Llama-3-8B-Instruct, Qwen2.5-Math, and Mistral-7B-Instruct). Our results unveil a contrast in final answer accuracy, where one model achieves near-human reliability while others cluster around substantially lower accuracy scores. In order to probe these discrepancies, we analyze step distributions of model responses and contrast them with the intrinsic structure of GSM8K. Although the dataset is naturally centered on succinct 2 to 4 step solutions, the model responses show varying tendencies: some systematically inflate reasoning chains beyond the required step count, while others collapse prematurely without providing a final numeric response. The verbose and under-reasoning failure modes are highlighted by these distributions, demonstrating that accuracy by itself is unable to fully convey the subtleties of LLMs' mathematical reasoning abilities. Together, our results highlight the potential and drawbacks of existing methods, showing that accurate mathematical reasoning necessitates alignment with the inherent cognitive problem structures.