{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/powerpropagation-a-sparsity-inducing-weight","title":"Powerpropagation: A sparsity inducing weight reparameterisation","arxiv_id":"2110.00296","date":"2021-10-01","proceeding":"NeurIPS 2021 12","authors":["Jonathan Schwarz","Siddhant M. Jayakumar","Razvan Pascanu","Peter E. Latham","Yee Whye Teh"],"abstract":"The training of sparse neural networks is becoming an increasingly important tool for reducing the computational footprint of models at training and evaluation, as well enabling the effective scaling up of models. Whereas much work over the years has been dedicated to specialised pruning techniques, little attention has been paid to the inherent effect of gradient based training on model sparsity. In this work, we introduce Powerpropagation, a new weight-parameterisation for neural networks that leads to inherently sparse models. Exploiting the behaviour of gradient descent, our method gives rise to weight updates exhibiting a \"rich get richer\" dynamic, leaving low-magnitude parameters largely unaffected by learning. Models trained in this manner exhibit similar performance, but have a distribution with markedly higher density at zero, allowing more parameters to be pruned safely. Powerpropagation is general, intuitive, cheap and straight-forward to implement and can readily be combined with various other techniques. To highlight its versatility, we explore it in two very different settings: Firstly, following a recent line of work, we investigate its effect on sparse training for resource-constrained settings. Here, we combine Powerpropagation with a traditional weight-pruning technique as well as recent state-of-the-art sparse-to-sparse algorithms, showing superior performance on the ImageNet benchmark. Secondly, we advocate the use of sparsity in overcoming catastrophic forgetting, where compressed representations allow accommodating a large number of tasks at fixed model capacity. In all cases our reparameterisation considerably increases the efficacy of the off-the-shelf methods.","url_abs":"https://arxiv.org/abs/2110.00296v2","url_pdf":"https://arxiv.org/pdf/2110.00296v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"powerpropagation-a-sparsity-inducing-weight","repo_url":"https://github.com/deepmind/deepmind-research/tree/master/powerpropagation","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"jax","reach":null},{"paper_slug":"powerpropagation-a-sparsity-inducing-weight","repo_url":"https://github.com/mysistinechapel/powerprop","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"powerpropagation","method_name":"Powerpropagation"},{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[{"slug":"powerpropagation","name":"Powerpropagation","full_name":"Powerpropagation"}],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2110.00296","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.00296"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/zajaczajac/cl-benchmark-tf2","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deepmind/deepmind-research/tree/master/powerpropagation","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mysistinechapel/powerprop","reach":null}],"summary":{"ran":1},"by_repo_kind":{"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"171cee6144d31993","entry":"PowerPropLinear","repo":"mysistinechapel/powerprop","repo_kind":"listed","path":"utils/pp_modules.py","file_url":"https://github.com/mysistinechapel/powerprop/blob/HEAD/utils/pp_modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"171cee6144d31993"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}