{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-sparse-neural-networks-through-l_0","title":"Learning Sparse Neural Networks through $L_0$ Regularization","arxiv_id":"1712.01312","date":"2017-12-04","proceeding":null,"authors":["Christos Louizos","Max Welling","Diederik P. Kingma"],"abstract":"We propose a practical method for $L_0$ norm regularization for neural\nnetworks: pruning the network during training by encouraging weights to become\nexactly zero. Such regularization is interesting since (1) it can greatly speed\nup training and inference, and (2) it can improve generalization. AIC and BIC,\nwell-known model selection criteria, are special cases of $L_0$ regularization.\nHowever, since the $L_0$ norm of weights is non-differentiable, we cannot\nincorporate it directly as a regularization term in the objective function. We\npropose a solution through the inclusion of a collection of non-negative\nstochastic gates, which collectively determine which weights to set to zero. We\nshow that, somewhat surprisingly, for certain distributions over the gates, the\nexpected $L_0$ norm of the resulting gated weights is differentiable with\nrespect to the distribution parameters. We further propose the \\emph{hard\nconcrete} distribution for the gates, which is obtained by \"stretching\" a\nbinary concrete distribution and then transforming its samples with a\nhard-sigmoid. The parameters of the distribution over the gates can then be\njointly optimized with the original network parameters. As a result our method\nallows for straightforward and efficient learning of model structures with\nstochastic gradient descent and allows for conditional computation in a\nprincipled way. We perform various experiments to demonstrate the effectiveness\nof the resulting approach and regularizer.","url_abs":"http://arxiv.org/abs/1712.01312v2","url_pdf":"http://arxiv.org/pdf/1712.01312v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-sparse-neural-networks-through-l_0","repo_url":"https://github.com/AMLab-Amsterdam/L0_regularization","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"learning-sparse-neural-networks-through-l_0","repo_url":"https://github.com/Soldie/stux-DNN","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"learning-sparse-neural-networks-through-l_0","repo_url":"https://github.com/bryankim96/stux-DNN","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"learning-sparse-neural-networks-through-l_0","repo_url":"https://github.com/ksheng-/sparse-nn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"model-selection","task_name":"Model Selection"}],"methods":[{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1712.01312","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}