Papers › Nonparametric Learning of Two-Layer ReLU Residual Units
Nonparametric Learning of Two-Layer ReLU Residual Units
Zhunxuan Wang, Linyun He, Chunchuan Lyu, Shay B. Cohen
We describe an algorithm that learns two-layer residual units using rectified linear unit (ReLU) activation: suppose the input 𝐱 is from a distribution with support space ℝᵈ and the ground-truth generative model is a residual unit of this type, given by 𝐲 = B^∗[(A^∗𝐱)^+ + 𝐱], where ground-truth network parameters A^∗ ∈ℝ^(d×d) represent a full-rank matrix with nonnegative entries and B^∗ ∈ℝ^(m×d) is full-rank with m ≥d and for c ∈ℝᵈ, [c⁺]ᵢ = max{0, cᵢ}. We design layer-wise objectives as functionals whose analytic minimizers express the exact ground-truth network in terms of its parameters and nonlinearities. Following this objective landscape, learning residual units from finite samples can be formulated using convex optimization of a nonparametric function: for each layer, we first formulate the corresponding empirical risk minimization (ERM) as a positive semi-definite quadratic program (QP), then we show the solution space of the QP can be equivalently determined by a set of linear inequalities, which can then be efficiently solved by linear programming (LP). We further prove the strong statistical consistency of our algorithm, and demonstrate its robustness and sample efficiency through experimental results on synthetic data and a set of benchmark regression datasets.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
No leaderboard rows for this paper in the archive.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections