Papers › Nonparametric Learning of Two-Layer ReLU Residual Units

Nonparametric Learning of Two-Layer ReLU Residual Units

17 Aug 2020arXiv:2008.07648archive 2025-07-28

Zhunxuan Wang, Linyun He, Chunchuan Lyu, Shay B. Cohen

We describe an algorithm that learns two-layer residual units using rectified linear unit (ReLU) activation: suppose the input x is from a distribution with support space ℝᵈ and the ground-truth generative model is a residual unit of this type, given by y = B^∗[(A^∗x)^+ + x], where ground-truth network parameters A^∗ ∈ℝ^(d×d) represent a full-rank matrix with nonnegative entries and B^∗ ∈ℝ^(m×d) is full-rank with m ≥d and for c ∈ℝᵈ, [c⁺]ᵢ = max{0, cᵢ}. We design layer-wise objectives as functionals whose analytic minimizers express the exact ground-truth network in terms of its parameters and nonlinearities. Following this objective landscape, learning residual units from finite samples can be formulated using convex optimization of a nonparametric function: for each layer, we first formulate the corresponding empirical risk minimization (ERM) as a positive semi-definite quadratic program (QP), then we show the solution space of the QP can be equivalently determined by a set of linear inequalities, which can then be efficiently solved by linear programming (LP). We further prove the strong statistical consistency of our algorithm, and demonstrate its robustness and sample efficiency through experimental results on synthetic data and a set of benchmark regression datasets.

PaperPDFConference PDFCode

Syntology This paper's arXiv id is not in Syntology's Atlas id list (549,449 ids, as this build read them; the read time is in the build record), so this page links to no Syntology page for it. The list is not the whole graph; Syntology's feed shows papers as they enter the graph.

Code

uuzeeex/relu-resunit-learning officialmentioned in paperSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Vocal Bursts Valence Prediction

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Methods

ReLU

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections