{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/advancing-model-pruning-via-bi-level","title":"Advancing Model Pruning via Bi-level Optimization","arxiv_id":"2210.04092","date":"2022-10-08","proceeding":null,"authors":["Yihua Zhang","Yuguang Yao","Parikshit Ram","Pu Zhao","Tianlong Chen","Mingyi Hong","Yanzhi Wang","Sijia Liu"],"abstract":"The deployment constraints in practical applications necessitate the pruning of large-scale deep learning models, i.e., promoting their weight sparsity. As illustrated by the Lottery Ticket Hypothesis (LTH), pruning also has the potential of improving their generalization ability. At the core of LTH, iterative magnitude pruning (IMP) is the predominant pruning method to successfully find 'winning tickets'. Yet, the computation cost of IMP grows prohibitively as the targeted pruning ratio increases. To reduce the computation overhead, various efficient 'one-shot' pruning methods have been developed, but these schemes are usually unable to find winning tickets as good as IMP. This raises the question of how to close the gap between pruning accuracy and pruning efficiency? To tackle it, we pursue the algorithmic advancement of model pruning. Specifically, we formulate the pruning problem from a fresh and novel viewpoint, bi-level optimization (BLO). We show that the BLO interpretation provides a technically-grounded optimization base for an efficient implementation of the pruning-retraining learning paradigm used in IMP. We also show that the proposed bi-level optimization-oriented pruning method (termed BiP) is a special class of BLO problems with a bi-linear problem structure. By leveraging such bi-linearity, we theoretically show that BiP can be solved as easily as first-order optimization, thus inheriting the computation efficiency. Through extensive experiments on both structured and unstructured pruning with 5 model architectures and 4 data sets, we demonstrate that BiP can find better winning tickets than IMP in most cases, and is computationally as efficient as the one-shot pruning schemes, demonstrating 2-7 times speedup over IMP for the same level of model accuracy and sparsity.","url_abs":"https://arxiv.org/abs/2210.04092v4","url_pdf":"https://arxiv.org/pdf/2210.04092v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"advancing-model-pruning-via-bi-level","repo_url":"https://github.com/optml-group/bip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"model","task_name":"model"}],"methods":[{"method_slug":"base","method_name":"BASE"},{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.04092","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.04092"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/OPTML-Group/BiP","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/optml-group/bip","reach":{"status":"ok"}}],"summary":{"ran":1,"unverified":6},"by_repo_kind":{"official":{"samples":7,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"c18c93fce5dd9f44","entry":"ProgressMeter","repo":"optml-group/bip","repo_kind":"official","path":"trainer/bilevel.py","file_url":"https://github.com/optml-group/bip/blob/HEAD/trainer/bilevel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c18c93fce5dd9f44"}},{"code_sha256_prefix":"f696344847408d8c","entry":"freeze_vars","repo":"optml-group/bip","repo_kind":"official","path":"trainer/bilevel.py","file_url":"https://github.com/optml-group/bip/blob/HEAD/trainer/bilevel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f696344847408d8c"}},{"code_sha256_prefix":"70a864a5cf4c3bde","entry":"switch_to_bilevel","repo":"optml-group/bip","repo_kind":"official","path":"trainer/bilevel.py","file_url":"https://github.com/optml-group/bip/blob/HEAD/trainer/bilevel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"70a864a5cf4c3bde"}},{"code_sha256_prefix":"522ea35e0105460b","entry":"switch_to_finetune","repo":"optml-group/bip","repo_kind":"official","path":"trainer/bilevel.py","file_url":"https://github.com/optml-group/bip/blob/HEAD/trainer/bilevel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"522ea35e0105460b"}},{"code_sha256_prefix":"4d54a1c0f56a0055","entry":"switch_to_prune","repo":"optml-group/bip","repo_kind":"official","path":"trainer/bilevel.py","file_url":"https://github.com/optml-group/bip/blob/HEAD/trainer/bilevel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4d54a1c0f56a0055"}},{"code_sha256_prefix":"9e1301e5ca5aa8c5","entry":"train","repo":"optml-group/bip","repo_kind":"official","path":"trainer/bilevel.py","file_url":"https://github.com/optml-group/bip/blob/HEAD/trainer/bilevel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9e1301e5ca5aa8c5"}},{"code_sha256_prefix":"010e64c40e6f9a21","entry":"unfreeze_vars","repo":"optml-group/bip","repo_kind":"official","path":"trainer/bilevel.py","file_url":"https://github.com/optml-group/bip/blob/HEAD/trainer/bilevel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"010e64c40e6f9a21"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}