{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/good-subnetworks-provably-exist-pruning-via","title":"Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection","arxiv_id":"2003.01794","date":"2020-03-03","proceeding":null,"authors":["Mao Ye","Chengyue Gong","Lizhen Nie","Denny Zhou","Adam Klivans","Qiang Liu"],"abstract":"Recent empirical works show that large deep neural networks are often highly redundant and one can find much smaller subnetworks without a significant drop of accuracy. However, most existing methods of network pruning are empirical and heuristic, leaving it open whether good subnetworks provably exist, how to find them efficiently, and if network pruning can be provably better than direct training using gradient descent. We answer these problems positively by proposing a simple greedy selection approach for finding good subnetworks, which starts from an empty network and greedily adds important neurons from the large network. This differs from the existing methods based on backward elimination, which remove redundant neurons from the large network. Theoretically, applying the greedy selection strategy on sufficiently large {pre-trained} networks guarantees to find small subnetworks with lower loss than networks directly trained with gradient descent. Our results also apply to pruning randomly weighted networks. Practically, we improve prior arts of network pruning on learning compact neural architectures on ImageNet, including ResNet, MobilenetV2/V3, and ProxylessNet. Our theory and empirical results on MobileNet suggest that we should fine-tune the pruned subnetworks to leverage the information from the large model, instead of re-training from new random initialization as suggested in \\citet{liu2018rethinking}.","url_abs":"https://arxiv.org/abs/2003.01794v3","url_pdf":"https://arxiv.org/pdf/2003.01794v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"good-subnetworks-provably-exist-pruning-via","repo_url":"https://github.com/lushleaf/Network-Pruning-Greedy-Forward-Selection","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"network-pruning","task_name":"Network Pruning"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"bottleneck-residual-block","method_name":"Bottleneck Residual Block"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"kaiming-initialization","method_name":"Kaiming Initialization"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"pruning","method_name":"Pruning"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-block","method_name":"Residual Block"},{"method_slug":"residual-connection","method_name":"Residual Connection"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2003.01794","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2003.01794"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lushleaf/Network-Pruning-Greedy-Forward-Selection","reach":null}],"summary":{"ran":1,"ran_fixture":1,"ran_honours":1,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":7,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"9c594fdddb872b1e","entry":"AverageMeter","repo":"lushleaf/Network-Pruning-Greedy-Forward-Selection","repo_kind":"official","path":"mobile2_prune.py","file_url":"https://github.com/lushleaf/Network-Pruning-Greedy-Forward-Selection/blob/HEAD/mobile2_prune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9c594fdddb872b1e"}},{"code_sha256_prefix":"30c2f02d39c9907a","entry":"accuracy","repo":"lushleaf/Network-Pruning-Greedy-Forward-Selection","repo_kind":"official","path":"mobile2_prune.py","file_url":"https://github.com/lushleaf/Network-Pruning-Greedy-Forward-Selection/blob/HEAD/mobile2_prune.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"30c2f02d39c9907a"}},{"code_sha256_prefix":"0e7a6eb592d6e2f5","entry":"adjust_learning_rate","repo":"lushleaf/Network-Pruning-Greedy-Forward-Selection","repo_kind":"official","path":"mobile2_finetune.py","file_url":"https://github.com/lushleaf/Network-Pruning-Greedy-Forward-Selection/blob/HEAD/mobile2_finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0e7a6eb592d6e2f5"}},{"code_sha256_prefix":"bbf80c3074fc4e23","entry":"decide_candidate_set","repo":"lushleaf/Network-Pruning-Greedy-Forward-Selection","repo_kind":"official","path":"mobile2_prune.py","file_url":"https://github.com/lushleaf/Network-Pruning-Greedy-Forward-Selection/blob/HEAD/mobile2_prune.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bbf80c3074fc4e23"}},{"code_sha256_prefix":"f774c5565db1ebc2","entry":"decide_candidate","repo":"lushleaf/Network-Pruning-Greedy-Forward-Selection","repo_kind":"official","path":"mobile2_prune.py","file_url":"https://github.com/lushleaf/Network-Pruning-Greedy-Forward-Selection/blob/HEAD/mobile2_prune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f774c5565db1ebc2"}},{"code_sha256_prefix":"a2264353ce143e03","entry":"eval_train","repo":"lushleaf/Network-Pruning-Greedy-Forward-Selection","repo_kind":"official","path":"mobile2_prune.py","file_url":"https://github.com/lushleaf/Network-Pruning-Greedy-Forward-Selection/blob/HEAD/mobile2_prune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a2264353ce143e03"}},{"code_sha256_prefix":"c29f2bdd7cf10d4d","entry":"prune_a_layer","repo":"lushleaf/Network-Pruning-Greedy-Forward-Selection","repo_kind":"official","path":"mobile2_prune.py","file_url":"https://github.com/lushleaf/Network-Pruning-Greedy-Forward-Selection/blob/HEAD/mobile2_prune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c29f2bdd7cf10d4d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}