{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-three-regime-model-of-network-pruning","title":"A Three-regime Model of Network Pruning","arxiv_id":"2305.18383","date":"2023-05-28","proceeding":null,"authors":["Yefan Zhou","Yaoqing Yang","Arin Chang","Michael W. Mahoney"],"abstract":"Recent work has highlighted the complex influence training hyperparameters, e.g., the number of training epochs, can have on the prunability of machine learning models. Perhaps surprisingly, a systematic approach to predict precisely how adjusting a specific hyperparameter will affect prunability remains elusive. To address this gap, we introduce a phenomenological model grounded in the statistical mechanics of learning. Our approach uses temperature-like and load-like parameters to model the impact of neural network (NN) training hyperparameters on pruning performance. A key empirical result we identify is a sharp transition phenomenon: depending on the value of a load-like parameter in the pruned model, increasing the value of a temperature-like parameter in the pre-pruned model may either enhance or impair subsequent pruning performance. Based on this transition, we build a three-regime model by taxonomizing the global structure of the pruned NN loss landscape. Our model reveals that the dichotomous effect of high temperature is associated with transitions between distinct types of global structures in the post-pruned model. Based on our results, we present three case-studies: 1) determining whether to increase or decrease a hyperparameter for improved pruning; 2) selecting the best model to prune from a family of models; and 3) tuning the hyperparameter of the Sharpness Aware Minimization method for better pruning performance.","url_abs":"https://arxiv.org/abs/2305.18383v1","url_pdf":"https://arxiv.org/pdf/2305.18383v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-three-regime-model-of-network-pruning","repo_url":"https://github.com/yefanzhou/threeregimepruning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"efficient-neural-network","task_name":"Efficient Neural Network"},{"task_slug":"hyperparameter-optimization","task_name":"Hyperparameter Optimization"},{"task_slug":"network-pruning","task_name":"Network Pruning"},{"task_slug":"model","task_name":"model"}],"methods":[{"method_slug":"aware","method_name":"AWARE"},{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.18383","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.18383"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/YefanZhou/ThreeRegimePruning","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yefanzhou/threeregimepruning","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran_draft_wrong":3,"ran_violates":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"e6423db5a218aa3f","entry":"_extract_weight_tuples","repo":"yefanzhou/threeregimepruning","repo_kind":"official","path":"src/three_regime_taxonomy/layer_adaptive_sparsity/pruners.py","file_url":"https://github.com/yefanzhou/threeregimepruning/blob/HEAD/src/three_regime_taxonomy/layer_adaptive_sparsity/pruners.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e6423db5a218aa3f"}},{"code_sha256_prefix":"e74bdd6268c72a4a","entry":"_is_prunable_module","repo":"yefanzhou/threeregimepruning","repo_kind":"official","path":"src/three_regime_taxonomy/layer_adaptive_sparsity/pruners.py","file_url":"https://github.com/yefanzhou/threeregimepruning/blob/HEAD/src/three_regime_taxonomy/layer_adaptive_sparsity/pruners.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e74bdd6268c72a4a"}},{"code_sha256_prefix":"f1b24d756f6ff408","entry":"get_modules","repo":"yefanzhou/threeregimepruning","repo_kind":"official","path":"src/three_regime_taxonomy/layer_adaptive_sparsity/pruners.py","file_url":"https://github.com/yefanzhou/threeregimepruning/blob/HEAD/src/three_regime_taxonomy/layer_adaptive_sparsity/pruners.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f1b24d756f6ff408"}},{"code_sha256_prefix":"9d5cecd63e0f976c","entry":"prune_weights_global_unstructured","repo":"yefanzhou/threeregimepruning","repo_kind":"official","path":"src/three_regime_taxonomy/layer_adaptive_sparsity/pruners.py","file_url":"https://github.com/yefanzhou/threeregimepruning/blob/HEAD/src/three_regime_taxonomy/layer_adaptive_sparsity/pruners.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9d5cecd63e0f976c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}