{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/progressive-skeletonization-trimming-more-fat","title":"Progressive Skeletonization: Trimming more fat from a network at initialization","arxiv_id":"2006.09081","date":"2020-06-16","proceeding":"ICLR 2021 1","authors":["Pau de Jorge","Amartya Sanyal","Harkirat S. Behl","Philip H. S. Torr","Gregory Rogez","Puneet K. Dokania"],"abstract":"Recent studies have shown that skeletonization (pruning parameters) of networks \\textit{at initialization} provides all the practical benefits of sparsity both at inference and training time, while only marginally degrading their performance. However, we observe that beyond a certain level of sparsity (approx $95\\%$), these approaches fail to preserve the network performance, and to our surprise, in many cases perform even worse than trivial random pruning. To this end, we propose an objective to find a skeletonized network with maximum {\\em foresight connection sensitivity} (FORCE) whereby the trainability, in terms of connection sensitivity, of a pruned network is taken into consideration. We then propose two approximate procedures to maximize our objective (1) Iterative SNIP: allows parameters that were unimportant at earlier stages of skeletonization to become important at later stages; and (2) FORCE: iterative process that allows exploration by allowing already pruned parameters to resurrect at later stages of skeletonization. Empirical analyses on a large suite of experiments show that our approach, while providing at least as good a performance as other recent approaches on moderate pruning levels, provides remarkably improved performance on higher pruning levels (could remove up to $99.5\\%$ parameters while keeping the networks trainable). Code can be found in https://github.com/naver/force.","url_abs":"https://arxiv.org/abs/2006.09081v5","url_pdf":"https://arxiv.org/pdf/2006.09081v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"progressive-skeletonization-trimming-more-fat","repo_url":"https://github.com/naver/force","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"sensitivity","task_name":"Sensitivity"}],"methods":[{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2006.09081","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.09081"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/naver/force","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":14},"by_repo_kind":{"official":{"samples":14,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9b8289076669fe4f","entry":"accuracy","repo":"naver/force","repo_kind":"official","path":"train_imagenet.py","file_url":"https://github.com/naver/force/blob/HEAD/train_imagenet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9b8289076669fe4f"}},{"code_sha256_prefix":"edda8bc5fb1dfa11","entry":"apply_prune_mask","repo":"naver/force","repo_kind":"official","path":"pruning/mask_networks.py","file_url":"https://github.com/naver/force/blob/HEAD/pruning/mask_networks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"edda8bc5fb1dfa11"}},{"code_sha256_prefix":"c2fe2e0433a97c2a","entry":"get_average_gradients","repo":"naver/force","repo_kind":"official","path":"pruning/pruning_algos.py","file_url":"https://github.com/naver/force/blob/HEAD/pruning/pruning_algos.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c2fe2e0433a97c2a"}},{"code_sha256_prefix":"a3aae03238435730","entry":"get_average_saliencies","repo":"naver/force","repo_kind":"official","path":"pruning/pruning_algos.py","file_url":"https://github.com/naver/force/blob/HEAD/pruning/pruning_algos.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a3aae03238435730"}},{"code_sha256_prefix":"b6cbcb7ccb3e0a51","entry":"get_cifar_test_loader","repo":"naver/force","repo_kind":"official","path":"experiments/datasets.py","file_url":"https://github.com/naver/force/blob/HEAD/experiments/datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b6cbcb7ccb3e0a51"}},{"code_sha256_prefix":"24854d716b08e3a5","entry":"get_cifar_train_valid_loader","repo":"naver/force","repo_kind":"official","path":"experiments/datasets.py","file_url":"https://github.com/naver/force/blob/HEAD/experiments/datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"24854d716b08e3a5"}},{"code_sha256_prefix":"3f216712b9e01cbb","entry":"get_mask","repo":"naver/force","repo_kind":"official","path":"pruning/pruning_algos.py","file_url":"https://github.com/naver/force/blob/HEAD/pruning/pruning_algos.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3f216712b9e01cbb"}},{"code_sha256_prefix":"e5ca87502d56226e","entry":"get_tiny_imagenet_train_valid_loader","repo":"naver/force","repo_kind":"official","path":"experiments/datasets.py","file_url":"https://github.com/naver/force/blob/HEAD/experiments/datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e5ca87502d56226e"}},{"code_sha256_prefix":"0327078528100688","entry":"resnet18","repo":"naver/force","repo_kind":"official","path":"experiments/models.py","file_url":"https://github.com/naver/force/blob/HEAD/experiments/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0327078528100688"}},{"code_sha256_prefix":"67cc5d9a1cd2e972","entry":"resnet34","repo":"naver/force","repo_kind":"official","path":"experiments/models.py","file_url":"https://github.com/naver/force/blob/HEAD/experiments/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"67cc5d9a1cd2e972"}},{"code_sha256_prefix":"881b8281d25db1b8","entry":"resnet50","repo":"naver/force","repo_kind":"official","path":"experiments/models.py","file_url":"https://github.com/naver/force/blob/HEAD/experiments/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"881b8281d25db1b8"}},{"code_sha256_prefix":"62eead0439dcfbc8","entry":"validate","repo":"naver/force","repo_kind":"official","path":"train_imagenet.py","file_url":"https://github.com/naver/force/blob/HEAD/train_imagenet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"62eead0439dcfbc8"}},{"code_sha256_prefix":"1cbecd8ff404af70","entry":"vgg_cifar_experiment","repo":"naver/force","repo_kind":"official","path":"experiments/experiments.py","file_url":"https://github.com/naver/force/blob/HEAD/experiments/experiments.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1cbecd8ff404af70"}},{"code_sha256_prefix":"472a885200c16eb7","entry":"vgg_tiny_imagenet_experiment","repo":"naver/force","repo_kind":"official","path":"experiments/experiments.py","file_url":"https://github.com/naver/force/blob/HEAD/experiments/experiments.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"472a885200c16eb7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}