The Zero Pattern of a Design Matrix Drives Multiple Descent in Over-parameterized Regression

Özetle: Over-parameterized linear regression has been widely studied over the last decade.

Özet

Over-parameterized linear regression has been widely studied over the last decade. However, most existing works assume that the covariates are independent and that their covariance matrices are non-degenerate. In this paper, we relax both assumptions and derive deterministic equivalents for the prediction risk in a vanishing-ridge regime. We show that degeneracy of the covariance matrices and dependence can lead to multiple descent, and characterize where the corresponding peaks can occur. Our proofs use a novel graph representation of the variance profile. We show that maximum matchings and the Dulmage–Mendelsohn decomposition of the associated bipartite graph identify the configurations at which the variance becomes singular.

Orijinal Özet (İngilizce)

Over-parameterized linear regression has been widely studied over the last decade. However, most existing works assume that the covariates are independent and that their covariance matrices are non-degenerate. In this paper, we relax both assumptions and derive deterministic equivalents for the prediction risk in a vanishing-ridge regime. We show that degeneracy of the covariance matrices and dependence can lead to multiple descent, and characterize where the corresponding peaks can occur. Our proofs use a novel graph representation of the variance profile. We show that maximum matchings and the Dulmage–Mendelsohn decomposition of the associated bipartite graph identify the configurations at which the variance becomes singular.

Kaynak: arXiv:2607.24041 · PDF