{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/compressed-deep-networks-goodbye-svd-hello","title":"Compressed Deep Networks: Goodbye SVD, Hello Robust Low-Rank Approximation","arxiv_id":"2009.05647","date":"2020-09-11","proceeding":null,"authors":["Murad Tukan","Alaa Maalouf","Matan Weksler","Dan Feldman"],"abstract":"A common technique for compressing a neural network is to compute the $k$-rank $\\ell_2$ approximation $A_{k,2}$ of the matrix $A\\in\\mathbb{R}^{n\\times d}$ that corresponds to a fully connected layer (or embedding layer). Here, $d$ is the number of the neurons in the layer, $n$ is the number in the next one, and $A_{k,2}$ can be stored in $O((n+d)k)$ memory instead of $O(nd)$. This $\\ell_2$-approximation minimizes the sum over every entry to the power of $p=2$ in the matrix $A - A_{k,2}$, among every matrix $A_{k,2}\\in\\mathbb{R}^{n\\times d}$ whose rank is $k$. While it can be computed efficiently via SVD, the $\\ell_2$-approximation is known to be very sensitive to outliers (\"far-away\" rows). Hence, machine learning uses e.g. Lasso Regression, $\\ell_1$-regularization, and $\\ell_1$-SVM that use the $\\ell_1$-norm. This paper suggests to replace the $k$-rank $\\ell_2$ approximation by $\\ell_p$, for $p\\in [1,2]$. We then provide practical and provable approximation algorithms to compute it for any $p\\geq1$, based on modern techniques in computational geometry. Extensive experimental results on the GLUE benchmark for compressing BERT, DistilBERT, XLNet, and RoBERTa confirm this theoretical advantage. For example, our approach achieves $28\\%$ compression of RoBERTa's embedding layer with only $0.63\\%$ additive drop in the accuracy (without fine-tuning) in average over all tasks in GLUE, compared to $11\\%$ drop using the existing $\\ell_2$-approximation. Open code is provided for reproducing and extending our results.","url_abs":"https://arxiv.org/abs/2009.05647v2","url_pdf":"https://arxiv.org/pdf/2009.05647v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"compressed-deep-networks-goodbye-svd-hello","repo_url":"https://github.com/muradtuk/LzModelCompression","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"distilbert","method_name":"DistilBERT"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"roberta","method_name":"RoBERTa"},{"method_slug":"sentencepiece","method_name":"SentencePiece"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"},{"method_slug":"xlnet","method_name":"XLNet"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2009.05647","atlas_url":"https://app.syntology.ai/?focus=2009.05647","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}