Spin Glass Analysis of Neural Network Training

Does a neural network’s loss landscape behave like a spin glass? Training a small MLP on a spiral task, this reads the diagonal of the Hessian as a spectrum of parameter-axis curvatures and follows how it sharpens epoch by epoch. The inverse participation ratio (IPR) measures how concentrated that curvature becomes — a finite-size echo of replica-symmetry breaking — tracing a glassy, broad-curvature start into a crystallised minimum, and tying the spin-glass ideas behind the 2024 Physics Nobel to ordinary network training.