{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deep-rewiring-training-very-sparse-deep","title":"Deep Rewiring: Training very sparse deep networks","arxiv_id":"1711.05136","date":"2017-11-14","proceeding":"ICLR 2018 1","authors":["Guillaume Bellec","David Kappel","Wolfgang Maass","Robert Legenstein"],"abstract":"Neuromorphic hardware tends to pose limits on the connectivity of deep\nnetworks that one can run on them. But also generic hardware and software\nimplementations of deep learning run more efficiently for sparse networks.\nSeveral methods exist for pruning connections of a neural network after it was\ntrained without connectivity constraints. We present an algorithm, DEEP R, that\nenables us to train directly a sparsely connected neural network. DEEP R\nautomatically rewires the network during supervised training so that\nconnections are there where they are most needed for the task, while its total\nnumber is all the time strictly bounded. We demonstrate that DEEP R can be used\nto train very sparse feedforward and recurrent neural networks on standard\nbenchmark tasks with just a minor loss in performance. DEEP R is based on a\nrigorous theoretical foundation that views rewiring as stochastic sampling of\nnetwork configurations from a posterior.","url_abs":"http://arxiv.org/abs/1711.05136v5","url_pdf":"http://arxiv.org/pdf/1711.05136v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deep-rewiring-training-very-sparse-deep","repo_url":"https://github.com/BITCS-Information-Retrieval-2020/search-low-levelmasterdosearch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"deep-rewiring-training-very-sparse-deep","repo_url":"https://github.com/IGITUGraz/LSNN-official","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"deep-rewiring-training-very-sparse-deep","repo_url":"https://github.com/guillaumeBellec/deep_rewiring","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"deep-rewiring-training-very-sparse-deep","repo_url":"https://gitlab.com/anon-dynamic-reparam/iclr2019-dynamic-reparam","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1711.05136","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}