arXiv:2604.03146v2 Announce Type: replace-cross
Abstract: We study high-dimensional convex empirical risk minimization (ERM) under general non-Gaussian data designs. By heuristically extending the Convex Gaussian Min-Max Theorem (CGMT) to non-Gaussian settings, we derive an asymptotic min-max characterization of key statistics, enabling approximation of the mean $mu_{hat{theta}}$ and covariance $C_{hat{theta}}$ of the ERM estimator $hat{theta}$.
Specifically, under a concentration assumption on the data matrix and standard regularity conditions on the loss and regularizer, we show that for a test covariate $x$ independent of the training data, the projection $hat{theta}^top x$ approximately follows the convolution of the generally non-Gaussian distribution of $mu_{hat{theta}}^top x$ with an independent centered Gaussian variable of variance $mathrm{tr}(C_{hat{theta}} mathbb{E}[xx^top])$. This result clarifies the scope and limits of Gaussian universality for ERMs.
Additionally, we prove that any $mathcal{C}^2$ regularizer is asymptotically equivalent to a quadratic form determined solely by its Hessian at zero and gradient at $mu_{hat{theta}}$.
Numerical simulations across diverse losses and models are provided to validate our theoretical predictions and qualitative insights.