We establish central and non-central limit theorems for sequences of geometric functionals of the limiting Gaussian output of random neural networks on the sphere. We show that, as the depth increases, the asymptotic behaviour is determined by the fixed points of the covariance kernel and leads to three possible regimes: convergence to the same functional evaluated at a limiting Gaussian field; convergence to a Gaussian distribution; or convergence to a spherical Rosenblatt/Hermite-type distribution.

More generally, we prove that the transition between these behaviours is governed by the uniform order of integrability (up to controlled errors) of the renormalized covariance function. This mechanism is closely related to what occurs for Gaussian fields with regularly varying covariances at infinity in the Euclidean setting, and reveals an analogous structure on the sphere. Based on a joint work with S. Di Lillo and D. Marinucci.