The Expectation--Maximization (EM) algorithm is a standard method for parameter estimation in Gaussian mixture models (GMMs). However, in high-dimensional settings with limited data, the classical EM algorithm often suffers from numerical instabilities due to ill-conditioned or singular covariance estimates. To address this issue, regularized variants of EM often introduce shrinkage of covariance matrices toward predefined target structures. Yet, their performance remains highly sensitive to the choice of the regularization parameter. In this paper, we propose a new gradient-based approach to adaptively update the regularization parameter during the iterations of the EM algorithm. Our method, denoted GMM-grad, leverages automatic differentiation to continuously adjust the penalization strength, in contrast to grid-search strategies. We provide a comprehensive empirical evaluation on both synthetic and real datasets, including text and image data, demonstrating that the proposed method achieves more stable and accurate clustering in challenging scenarios.