Papers › LDLT L-Lipschitz Network Weight Parameterization Initialization
LDLT L-Lipschitz Network Weight Parameterization Initialization
Marius F. R. Juston, Ramavarapu S. Sreenivas, Dustin Nottage, Ahmet Soylemezoglu
Title, abstract, authors and date from arXiv's metadata (CC0); this paper is not in the Papers with Code archive (frozen 2025-07-28).
We analyze initialization dynamics for LDLT-based ℒ-Lipschitz layers by deriving the exact marginal output variance when the underlying parameter matrix W₀∈ℝ^(m×n) is initialized with IID Gaussian entries 𝒩(0,σ²). The Wishart distribution, S=W₀W₀^⊤∼𝒲ₘ(n,σ² Iₘ), used for computing the output marginal variance is derived in closed form using expectations of zonal polynomials via James' theorem and a Laplace-integral expansion of (αIₘ+S)⁻¹. We develop an Isserlis/Wick-based combinatorial expansion for 𝔼[tr(Sᵏ)] and provide explicit truncated moments up to k=10, which yield accurate series approximations for small-to-moderate σ². Monte Carlo experiments confirm the theoretical estimates. Furthermore, empirical analysis was performed to quantify that, using current He or Kaiming initialization with scaling 1/√(n), the output variance is $0.41$, whereas the new parameterization with 10/ √(n) for α=1 results in an output variance of $0.9$. The findings clarify why deep ℒ-Lipschitz networks suffer rapid information loss at initialization and offer practical prescriptions for choosing initialization hyperparameters to mitigate this effect. However, using the Higgs boson classification dataset, a hyperparameter sweep over optimizers, initialization scale, and depth was conducted to validate the results on real-world data, showing that although the derivation ensures variance preservation, empirical results indicate He initialization still performs better.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Syntology holds the repository link but has not harvested or run code from it.
Results from the paper
The Papers with Code archive ends with its 2025-07-28 snapshot. This paper's arXiv identifier, 2601.08253, was issued in January 2026, after that date, so the archive has no leaderboard rows for it.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections