Mind the Knowledge Gap: Evaluating AI Tutors' Ability to Detect Mathematical Prior Knowledge and Misconceptions

EasyChair Preprint 15939

Abstract

Designing effective mathematics AI tutors presents a unique challenge in the field of AI in Education. While it is possible to train an AI model with strong mathematical knowledge, the role of ‘AI as a tutor’, specifically, taking on a traditional tutor’s key role in reviewing students’ misconceptions and knowledge gaps, is an under-explored area. This creates a risk that many AI tutoring experiences cannot be adequately tailored to student needs. Consequently, students may develop persistent misconceptions that go unaddressed because AI systems lack the diagnostic capabilities to identify specific gaps in understanding. Despite the growing integration of AI in education, the mathematics education community has yet to adequately address the foundational tasks of defining and developing pedagogically sound technique to test the capabilities of AI models to correctly detect misconceptions as well as knowledge gaps students have or develop. Many AI-powered resources, including AI tutors, are developed by commercial companies designing large language models, whose priorities are often driven more by market trends and hype than by learning sciences (e.g., addressing the needs and characteristics of learners and teachers). In this paper, we envision a training process for the development of a high-quality mathematics AI tutor embedded with a constructivist approach (emphasising that learners actively construct their own knowledge rather than being passively guided by questions), which is an alternative to the commonly adapted Socratic dialogue, to facilitate effective self-directed mathematics learning. The model would focus on the very first part of the tutoring process, which is identifying student misconceptions and knowledge gaps. In this paper, we propose a technique to test the capabilities of an AI model to correctly detect misconceptions students have developed. This is done so that more personalised and effective tutoring can be achieved.

Keyphrases: AI Tutoring Systems, AI tutor, Constructivism, Intelligent Tutoring Systems, Personalised learning experience, Zone of Proximal Development, effective teaching, effective tutoring experience, knowledge and misconceptions, large language models, learning needs, mathematical reasoning, mathematical reasoning skills, mathematics education, misconceptions and knowledge gaps, personalised learning, perspective on digital learning, prior knowledge, student misconceptions and knowledge gaps, technique to test, tutor llm, two level interactive test

@booklet{EasyChair:15939,
  author    = {Xinyue Li and Frank Morley and Rachad Zaki},
  title     = {Mind the Knowledge Gap: Evaluating AI Tutors' Ability to Detect Mathematical Prior Knowledge and Misconceptions},
  howpublished = {EasyChair Preprint 15939},
  year      = {EasyChair, 2025}}