{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pruning-neural-networks-without-any-data-by","title":"Pruning neural networks without any data by iteratively conserving synaptic flow","arxiv_id":"2006.05467","date":"2020-06-09","proceeding":"NeurIPS 2020 12","authors":["Hidenori Tanaka","Daniel Kunin","Daniel L. K. Yamins","Surya Ganguli"],"abstract":"Pruning the parameters of deep neural networks has generated intense interest due to potential savings in time, memory and energy both during training and at test time. Recent works have identified, through an expensive sequence of training and pruning cycles, the existence of winning lottery tickets or sparse trainable subnetworks at initialization. This raises a foundational question: can we identify highly sparse trainable subnetworks at initialization, without ever training, or indeed without ever looking at the data? We provide an affirmative answer to this question through theory driven algorithm design. We first mathematically formulate and experimentally verify a conservation law that explains why existing gradient-based pruning algorithms at initialization suffer from layer-collapse, the premature pruning of an entire layer rendering a network untrainable. This theory also elucidates how layer-collapse can be entirely avoided, motivating a novel pruning algorithm Iterative Synaptic Flow Pruning (SynFlow). This algorithm can be interpreted as preserving the total flow of synaptic strengths through the network at initialization subject to a sparsity constraint. Notably, this algorithm makes no reference to the training data and consistently competes with or outperforms existing state-of-the-art pruning algorithms at initialization over a range of models (VGG and ResNet), datasets (CIFAR-10/100 and Tiny ImageNet), and sparsity constraints (up to 99.99 percent). Thus our data-agnostic pruning algorithm challenges the existing paradigm that, at initialization, data must be used to quantify which synapses are important.","url_abs":"https://arxiv.org/abs/2006.05467v3","url_pdf":"https://arxiv.org/pdf/2006.05467v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pruning-neural-networks-without-any-data-by","repo_url":"https://github.com/ganguli-lab/Synaptic-Flow","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"pruning-neural-networks-without-any-data-by","repo_url":"https://github.com/Hoonyyhoon/Comparison-of-Synflow_SNIP_GraSP","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"pruning-neural-networks-without-any-data-by","repo_url":"https://github.com/apd10/synaptic-flow","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"pruning-neural-networks-without-any-data-by","repo_url":"https://github.com/iurada/px-ntk-pruning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"pruning-neural-networks-without-any-data-by","repo_url":"https://github.com/pvh1602/npb","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"pruning-neural-networks-without-any-data-by","repo_url":"https://github.com/yitewang/ntk-sap","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2006.05467","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.05467"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ganguli-lab/Synaptic-Flow","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/apd10/synaptic-flow","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yitewang/ntk-sap","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/iurada/px-ntk-pruning","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Hoonyyhoon/Comparison-of-Synflow_SNIP_GraSP","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pvh1602/npb","reach":null}],"summary":{"ran":11,"ran_draft_wrong":1,"unverified":5},"by_repo_kind":{"listed":{"samples":17,"ran":12,"repositories":5}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":16,"samples":[{"code_sha256_prefix":"7935d0e889ff9eee","entry":"BatchNorm1d","repo":"iurada/px-ntk-pruning","repo_kind":"listed","path":"lib/pruners.py","file_url":"https://github.com/iurada/px-ntk-pruning/blob/HEAD/lib/pruners.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7935d0e889ff9eee"}},{"code_sha256_prefix":"bd2e9ef9ef047b34","entry":"BatchNorm2d","repo":"iurada/px-ntk-pruning","repo_kind":"listed","path":"lib/pruners.py","file_url":"https://github.com/iurada/px-ntk-pruning/blob/HEAD/lib/pruners.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bd2e9ef9ef047b34"}},{"code_sha256_prefix":"d526bce01cf543a6","entry":"Conv2d","repo":"iurada/px-ntk-pruning","repo_kind":"listed","path":"lib/pruners.py","file_url":"https://github.com/iurada/px-ntk-pruning/blob/HEAD/lib/pruners.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d526bce01cf543a6"}},{"code_sha256_prefix":"c09e99a71aca360c","entry":"Identity1d","repo":"iurada/px-ntk-pruning","repo_kind":"listed","path":"lib/pruners.py","file_url":"https://github.com/iurada/px-ntk-pruning/blob/HEAD/lib/pruners.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c09e99a71aca360c"}},{"code_sha256_prefix":"f3d19807c24c2f63","entry":"Identity2d","repo":"iurada/px-ntk-pruning","repo_kind":"listed","path":"lib/pruners.py","file_url":"https://github.com/iurada/px-ntk-pruning/blob/HEAD/lib/pruners.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f3d19807c24c2f63"}},{"code_sha256_prefix":"dc24efa85753be6b","entry":"Linear","repo":"iurada/px-ntk-pruning","repo_kind":"listed","path":"lib/pruners.py","file_url":"https://github.com/iurada/px-ntk-pruning/blob/HEAD/lib/pruners.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dc24efa85753be6b"}},{"code_sha256_prefix":"e28e1790d31fe600","entry":"Pruner","repo":"pvh1602/npb","repo_kind":"listed","path":"Pruners/pruners.py","file_url":"https://github.com/pvh1602/npb/blob/HEAD/Pruners/pruners.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e28e1790d31fe600"}},{"code_sha256_prefix":"80271ea817e0be5e","entry":"SynFlow","repo":"pvh1602/npb","repo_kind":"listed","path":"Pruners/pruners.py","file_url":"https://github.com/pvh1602/npb/blob/HEAD/Pruners/pruners.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"80271ea817e0be5e"}},{"code_sha256_prefix":"e7ca315e78c80c60","entry":"SynFlow","repo":"apd10/synaptic-flow","repo_kind":"listed","path":"Pruners/pruners.py","file_url":"https://github.com/apd10/synaptic-flow/blob/HEAD/Pruners/pruners.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e7ca315e78c80c60"}},{"code_sha256_prefix":"97a5db95987dac5b","entry":"SynFlow","repo":"yitewang/ntk-sap","repo_kind":"listed","path":"Pruners/pruners.py","file_url":"https://github.com/yitewang/ntk-sap/blob/HEAD/Pruners/pruners.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"97a5db95987dac5b"}},{"code_sha256_prefix":"772bf8519445187f","entry":"Synflow","repo":"Hoonyyhoon/Comparison-of-Synflow_SNIP_GraSP","repo_kind":"listed","path":"pruning_method/Synflow.py","file_url":"https://github.com/Hoonyyhoon/Comparison-of-Synflow_SNIP_GraSP/blob/HEAD/pruning_method/Synflow.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"772bf8519445187f"}},{"code_sha256_prefix":"b4642e3b07dce20e","entry":"masks","repo":"iurada/px-ntk-pruning","repo_kind":"listed","path":"lib/pruners.py","file_url":"https://github.com/iurada/px-ntk-pruning/blob/HEAD/lib/pruners.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b4642e3b07dce20e"}},{"code_sha256_prefix":"80f512af69686019","entry":"Pruner","repo":"Hoonyyhoon/Comparison-of-Synflow_SNIP_GraSP","repo_kind":"listed","path":"pruning_method/Synflow.py","file_url":"https://github.com/Hoonyyhoon/Comparison-of-Synflow_SNIP_GraSP/blob/HEAD/pruning_method/Synflow.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"80f512af69686019"}},{"code_sha256_prefix":"895223082b0c7453","entry":"Pruner","repo":"iurada/px-ntk-pruning","repo_kind":"listed","path":"lib/pruners.py","file_url":"https://github.com/iurada/px-ntk-pruning/blob/HEAD/lib/pruners.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"895223082b0c7453"}},{"code_sha256_prefix":"60c6b324705c2d2b","entry":"SynFlow","repo":"iurada/px-ntk-pruning","repo_kind":"listed","path":"lib/pruners.py","file_url":"https://github.com/iurada/px-ntk-pruning/blob/HEAD/lib/pruners.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"60c6b324705c2d2b"}},{"code_sha256_prefix":"3df869cac18fe082","entry":"masked_parameters","repo":"iurada/px-ntk-pruning","repo_kind":"listed","path":"lib/pruners.py","file_url":"https://github.com/iurada/px-ntk-pruning/blob/HEAD/lib/pruners.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3df869cac18fe082"}},{"code_sha256_prefix":"3127d6e8725313f3","entry":"prunable","repo":"iurada/px-ntk-pruning","repo_kind":"listed","path":"lib/pruners.py","file_url":"https://github.com/iurada/px-ntk-pruning/blob/HEAD/lib/pruners.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3127d6e8725313f3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}