Papers › Compressed Deep Networks: Goodbye SVD, Hello Robust Low-Rank Approximation
Compressed Deep Networks: Goodbye SVD, Hello Robust Low-Rank Approximation
Murad Tukan, Alaa Maalouf, Matan Weksler, Dan Feldman
A common technique for compressing a neural network is to compute the k-rank ℓ₂ approximation A_(k,2) of the matrix A∈ℝ^(n×d) that corresponds to a fully connected layer (or embedding layer). Here, d is the number of the neurons in the layer, n is the number in the next one, and A_(k,2) can be stored in O((n+d)k) memory instead of O(nd). This ℓ₂-approximation minimizes the sum over every entry to the power of p=2 in the matrix A - A_(k,2), among every matrix A_(k,2)∈ℝ^(n×d) whose rank is k. While it can be computed efficiently via SVD, the ℓ₂-approximation is known to be very sensitive to outliers ("far-away" rows). Hence, machine learning uses e.g. Lasso Regression, ℓ₁-regularization, and ℓ₁-SVM that use the ℓ₁-norm. This paper suggests to replace the k-rank ℓ₂ approximation by ℓₚ, for p∈[1,2]. We then provide practical and provable approximation algorithms to compute it for any p≥1, based on modern techniques in computational geometry. Extensive experimental results on the GLUE benchmark for compressing BERT, DistilBERT, XLNet, and RoBERTa confirm this theoretical advantage. For example, our approach achieves 28% compression of RoBERTa's embedding layer with only 0.63% additive drop in the accuracy (without fine-tuning) in average over all tasks in GLUE, compared to 11% drop using the existing ℓ₂-approximation. Open code is provided for reproducing and extending our results.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Results from the paper archive 2025-07-28
No leaderboard rows for this paper in the archive.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections