Artificial intelligence in education is bringing significant changes to the way learning and teaching is done, and its application in mathematics teaching opens up numerous opportunities for improving the educational process. There is a clear connection between mathematics and artificial intelligence, as many AI algorithms and technologies base their functionalities on mathematical principles, such as statistics, linear algebra, and optimization theory. This article explores the mathematical foundations that support the development of artificial intelligence, emphasizing how mathematics provides the foundation for understanding and implementing AI technologies. The emphasis of the article will be on the application of artificial intelligence in mathematics teaching, with the aim of improving the experience of students and teachers. AI enables personalized learning, where mathematical models are used to adapt tasks and teaching content to the specific needs of each student. This research will explain how artificial intelligence can help optimize teaching methods, improve learning efficiency, and support the development of metacognitive skills in students. Although the advantages of applying artificial intelligence in education are significant, it is important to consider the challenges and obstacles that may be encountered in implementing this technology in order to fully utilize its potential in mathematics teaching.