{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/paying-more-attention-to-snapshots-of","title":"Paying more attention to snapshots of Iterative Pruning: Improving Model Compression via Ensemble Distillation","arxiv_id":"2006.11487","date":"2020-06-20","proceeding":"BMVC 2020 8","authors":["Duong H. Le","Trung-Nhan Vo","Nam Thoai"],"abstract":"Network pruning is one of the most dominant methods for reducing the heavy inference cost of deep neural networks. Existing methods often iteratively prune networks to attain high compression ratio without incurring significant loss in performance. However, we argue that conventional methods for retraining pruned networks (i.e., using small, fixed learning rate) are inadequate as they completely ignore the benefits from snapshots of iterative pruning. In this work, we show that strong ensembles can be constructed from snapshots of iterative pruning, which achieve competitive performance and vary in network structure. Furthermore, we present simple, general and effective pipeline that generates strong ensembles of networks during pruning with large learning rate restarting, and utilizes knowledge distillation with those ensembles to improve the predictive power of compact models. In standard image classification benchmarks such as CIFAR and Tiny-Imagenet, we advance state-of-the-art pruning ratio of structured pruning by integrating simple l1-norm filters pruning into our pipeline. Specifically, we reduce 75-80% of total parameters and 65-70% MACs of numerous variants of ResNet architectures while having comparable or better performance than that of original networks. Code associate with this paper is made publicly available at https://github.com/lehduong/kesi.","url_abs":"https://arxiv.org/abs/2006.11487v3","url_pdf":"https://arxiv.org/pdf/2006.11487v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"paying-more-attention-to-snapshots-of","repo_url":"https://github.com/lehduong/kesi","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"model-compression","task_name":"Model Compression"},{"task_slug":"network-pruning","task_name":"Network Pruning"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"bottleneck-residual-block","method_name":"Bottleneck Residual Block"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"kaiming-initialization","method_name":"Kaiming Initialization"},{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"pruning","method_name":"Pruning"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-block","method_name":"Residual Block"},{"method_slug":"residual-connection","method_name":"Residual Connection"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2006.11487","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.11487"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lehduong/kesi","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"unverified":7},"by_repo_kind":{"official":{"samples":8,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fac5364e2f53c6db","entry":"conv3x3","repo":"lehduong/kesi","repo_kind":"official","path":"cifar/filter_pruning/models/preresnet.py","file_url":"https://github.com/lehduong/kesi/blob/HEAD/cifar/filter_pruning/models/preresnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fac5364e2f53c6db"}},{"code_sha256_prefix":"4d3faaaa1c706b0b","entry":"accuracy","repo":"lehduong/kesi","repo_kind":"official","path":"imagenet/l1-norm-pruning/ensemble_finetune.py","file_url":"https://github.com/lehduong/kesi/blob/HEAD/imagenet/l1-norm-pruning/ensemble_finetune.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4d3faaaa1c706b0b"}},{"code_sha256_prefix":"80752954f2a868b8","entry":"accuracy","repo":"lehduong/kesi","repo_kind":"official","path":"imagenet/weight-level/main_finetune.py","file_url":"https://github.com/lehduong/kesi/blob/HEAD/imagenet/weight-level/main_finetune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"80752954f2a868b8"}},{"code_sha256_prefix":"7835366b1d8d0c63","entry":"create_l1_norm_mask","repo":"lehduong/kesi","repo_kind":"official","path":"imagenet/l1-norm-pruning/residualprune.py","file_url":"https://github.com/lehduong/kesi/blob/HEAD/imagenet/l1-norm-pruning/residualprune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7835366b1d8d0c63"}},{"code_sha256_prefix":"89d6cfa68bf25631","entry":"get_conv_zero_param","repo":"lehduong/kesi","repo_kind":"official","path":"imagenet/weight-level/main_finetune.py","file_url":"https://github.com/lehduong/kesi/blob/HEAD/imagenet/weight-level/main_finetune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"89d6cfa68bf25631"}},{"code_sha256_prefix":"15487f2b7b36c9cc","entry":"validate","repo":"lehduong/kesi","repo_kind":"official","path":"imagenet/l1-norm-pruning/ensemble_finetune.py","file_url":"https://github.com/lehduong/kesi/blob/HEAD/imagenet/l1-norm-pruning/ensemble_finetune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"15487f2b7b36c9cc"}},{"code_sha256_prefix":"1dcc89e293ab6570","entry":"validate","repo":"lehduong/kesi","repo_kind":"official","path":"imagenet/l1-norm-pruning/main_finetune.py","file_url":"https://github.com/lehduong/kesi/blob/HEAD/imagenet/l1-norm-pruning/main_finetune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1dcc89e293ab6570"}},{"code_sha256_prefix":"61e4d0bc645ebb64","entry":"validate","repo":"lehduong/kesi","repo_kind":"official","path":"imagenet/weight-level/main_finetune.py","file_url":"https://github.com/lehduong/kesi/blob/HEAD/imagenet/weight-level/main_finetune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"61e4d0bc645ebb64"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}