{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/net-trim-convex-pruning-of-deep-neural","title":"Net-Trim: Convex Pruning of Deep Neural Networks with Performance Guarantee","arxiv_id":"1611.05162","date":"2016-11-16","proceeding":"NeurIPS 2017 12","authors":["Alireza Aghasi","Afshin Abdi","Nam Nguyen","Justin Romberg"],"abstract":"We introduce and analyze a new technique for model reduction for deep neural\nnetworks. While large networks are theoretically capable of learning\narbitrarily complex models, overfitting and model redundancy negatively affects\nthe prediction accuracy and model variance. Our Net-Trim algorithm prunes\n(sparsifies) a trained network layer-wise, removing connections at each layer\nby solving a convex optimization program. This program seeks a sparse set of\nweights at each layer that keeps the layer inputs and outputs consistent with\nthe originally trained model. The algorithms and associated analysis are\napplicable to neural networks operating with the rectified linear unit (ReLU)\nas the nonlinear activation. We present both parallel and cascade versions of\nthe algorithm. While the latter can achieve slightly simpler models with the\nsame generalization performance, the former can be computed in a distributed\nmanner. In both cases, Net-Trim significantly reduces the number of connections\nin the network, while also providing enough regularization to slightly reduce\nthe generalization error. We also provide a mathematical analysis of the\nconsistency between the initial network and the retrained model. To analyze the\nmodel sample complexity, we derive the general sufficient conditions for the\nrecovery of a sparse transform matrix. For a single layer taking independent\nGaussian random vectors of length $N$ as inputs, we show that if the network\nresponse can be described using a maximum number of $s$ non-zero weights per\nnode, these weights can be learned from $\\mathcal{O}(s\\log N)$ samples.","url_abs":"http://arxiv.org/abs/1611.05162v4","url_pdf":"http://arxiv.org/pdf/1611.05162v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"net-trim-convex-pruning-of-deep-neural","repo_url":"https://github.com/DNNToolBox/Net-Trim-v1","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}