{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/network-pruning-that-matters-a-case-study-on-1","title":"Network Pruning That Matters: A Case Study on Retraining Variants","arxiv_id":"2105.03193","date":"2021-05-07","proceeding":"ICLR 2021 1","authors":["Duong H. Le","Binh-Son Hua"],"abstract":"Network pruning is an effective method to reduce the computational expense of over-parameterized neural networks for deployment on low-resource systems. Recent state-of-the-art techniques for retraining pruned networks such as weight rewinding and learning rate rewinding have been shown to outperform the traditional fine-tuning technique in recovering the lost accuracy (Renda et al., 2020), but so far it is unclear what accounts for such performance. In this work, we conduct extensive experiments to verify and analyze the uncanny effectiveness of learning rate rewinding. We find that the reason behind the success of learning rate rewinding is the usage of a large learning rate. Similar phenomenon can be observed in other learning rate schedules that involve large learning rates, e.g., the 1-cycle learning rate schedule (Smith et al., 2019). By leveraging the right learning rate schedule in retraining, we demonstrate a counter-intuitive phenomenon in that randomly pruned networks could even achieve better performance than methodically pruned networks (fine-tuned with the conventional approach). Our results emphasize the cruciality of the learning rate schedule in pruned network retraining - a detail often overlooked by practitioners during the implementation of network pruning. One-sentence Summary: We study the effective of different retraining mechanisms while doing pruning","url_abs":"https://arxiv.org/abs/2105.03193v1","url_pdf":"https://arxiv.org/pdf/2105.03193v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"network-pruning-that-matters-a-case-study-on-1","repo_url":"https://github.com/lehduong/NPTM","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"network-pruning","task_name":"Network Pruning"},{"task_slug":"sentence","task_name":"Sentence"}],"methods":[{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/network-pruning-on-imagenet","task":"Network Pruning","dataset":"ImageNet","model":"ResNet50","rank_in_archive_order":10,"of":16,"metrics":{"Accuracy":"75.59"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2105.03193","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2105.03193"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lehduong/NPTM","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"unverified":10},"by_repo_kind":{"official":{"samples":12,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fac5364e2f53c6db","entry":"conv3x3","repo":"lehduong/NPTM","repo_kind":"official","path":"hrank/models/resnet_imagenet.py","file_url":"https://github.com/lehduong/NPTM/blob/HEAD/hrank/models/resnet_imagenet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fac5364e2f53c6db"}},{"code_sha256_prefix":"583f9780bdd00a45","entry":"conv3x3","repo":"lehduong/NPTM","repo_kind":"official","path":"hrank/models/resnet_cifar.py","file_url":"https://github.com/lehduong/NPTM/blob/HEAD/hrank/models/resnet_cifar.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"583f9780bdd00a45"}},{"code_sha256_prefix":"48d66c6c1d311f14","entry":"adjust_learning_rate","repo":"lehduong/NPTM","repo_kind":"official","path":"soft-filter/pruning_cifar10_resnet.py","file_url":"https://github.com/lehduong/NPTM/blob/HEAD/soft-filter/pruning_cifar10_resnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"48d66c6c1d311f14"}},{"code_sha256_prefix":"35cdfb0ed0def06b","entry":"densenet_40","repo":"lehduong/NPTM","repo_kind":"official","path":"hrank/models/densenet_cifar.py","file_url":"https://github.com/lehduong/NPTM/blob/HEAD/hrank/models/densenet_cifar.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"35cdfb0ed0def06b"}},{"code_sha256_prefix":"76cc74efee42ba21","entry":"get_num_gen","repo":"lehduong/NPTM","repo_kind":"official","path":"hrank/get_flops.py","file_url":"https://github.com/lehduong/NPTM/blob/HEAD/hrank/get_flops.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"76cc74efee42ba21"}},{"code_sha256_prefix":"e1e2c11c83101616","entry":"googlenet","repo":"lehduong/NPTM","repo_kind":"official","path":"hrank/models/googlenet_cifar.py","file_url":"https://github.com/lehduong/NPTM/blob/HEAD/hrank/models/googlenet_cifar.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e1e2c11c83101616"}},{"code_sha256_prefix":"26ff085b343fa39e","entry":"is_leaf","repo":"lehduong/NPTM","repo_kind":"official","path":"hrank/get_flops.py","file_url":"https://github.com/lehduong/NPTM/blob/HEAD/hrank/get_flops.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"26ff085b343fa39e"}},{"code_sha256_prefix":"5c4dab079bf6276e","entry":"is_pruned","repo":"lehduong/NPTM","repo_kind":"official","path":"hrank/get_flops.py","file_url":"https://github.com/lehduong/NPTM/blob/HEAD/hrank/get_flops.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5c4dab079bf6276e"}},{"code_sha256_prefix":"ec44e217b9b585a3","entry":"resnet_110","repo":"lehduong/NPTM","repo_kind":"official","path":"hrank/models/resnet_cifar.py","file_url":"https://github.com/lehduong/NPTM/blob/HEAD/hrank/models/resnet_cifar.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec44e217b9b585a3"}},{"code_sha256_prefix":"858ef1991b5565f5","entry":"resnet_50","repo":"lehduong/NPTM","repo_kind":"official","path":"hrank/models/resnet_imagenet.py","file_url":"https://github.com/lehduong/NPTM/blob/HEAD/hrank/models/resnet_imagenet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"858ef1991b5565f5"}},{"code_sha256_prefix":"4df800028ee3d293","entry":"resnet_56","repo":"lehduong/NPTM","repo_kind":"official","path":"hrank/models/resnet_cifar.py","file_url":"https://github.com/lehduong/NPTM/blob/HEAD/hrank/models/resnet_cifar.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4df800028ee3d293"}},{"code_sha256_prefix":"60ce64be0e952e4b","entry":"vgg_16_bn","repo":"lehduong/NPTM","repo_kind":"official","path":"hrank/models/vgg.py","file_url":"https://github.com/lehduong/NPTM/blob/HEAD/hrank/models/vgg.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"60ce64be0e952e4b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}