{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/structured-pruning-learns-compact-and-1","title":"Structured Pruning Learns Compact and Accurate Models","arxiv_id":"2204.00408","date":"2022-04-01","proceeding":"ACL 2022 5","authors":["Mengzhou Xia","Zexuan Zhong","Danqi Chen"],"abstract":"The growing size of neural language models has led to increased attention in model compression. The two predominant approaches are pruning, which gradually removes weights from a pre-trained model, and distillation, which trains a smaller compact model to match a larger one. Pruning methods can significantly reduce the model size but hardly achieve large speedups as distillation. However, distillation methods require large amounts of unlabeled data and are expensive to train. In this work, we propose a task-specific structured pruning method CoFi (Coarse- and Fine-grained Pruning), which delivers highly parallelizable subnetworks and matches the distillation methods in both accuracy and latency, without resorting to any unlabeled data. Our key insight is to jointly prune coarse-grained (e.g., layers) and fine-grained (e.g., heads and hidden units) modules, which controls the pruning decision of each parameter with masks of different granularity. We also devise a layerwise distillation strategy to transfer knowledge from unpruned to pruned models during optimization. Our experiments on GLUE and SQuAD datasets show that CoFi yields models with over 10x speedups with a small accuracy drop, showing its effectiveness and efficiency compared to previous pruning and distillation approaches.","url_abs":"https://arxiv.org/abs/2204.00408v3","url_pdf":"https://arxiv.org/pdf/2204.00408v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"structured-pruning-learns-compact-and-1","repo_url":"https://github.com/princeton-nlp/cofipruning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"structured-pruning-learns-compact-and-1","repo_url":"https://github.com/jongwooko/nash-pruning-official","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"model-compression","task_name":"Model Compression"}],"methods":[{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2204.00408","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2204.00408"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jongwooko/nash-pruning-official","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/princeton-nlp/cofipruning","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/huawei-noah/Pretrained-Language-Model","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/neuralmagic/deepsparse","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":7,"ran_draft_wrong":2,"unverified":3},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":1,"ran":0,"repositories":1},"found_in_text":{"samples":10,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"ef9ec1a11223fbd0","entry":"L0Module","repo":"princeton-nlp/cofipruning","repo_kind":"official","path":"models/l0_module.py","file_url":"https://github.com/princeton-nlp/cofipruning/blob/HEAD/models/l0_module.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ef9ec1a11223fbd0"}},{"code_sha256_prefix":"f868beb1186a0be6","entry":"SuperBertIntermediate","repo":"huawei-noah/Pretrained-Language-Model","repo_kind":"found_in_text","path":"AutoTinyBERT/transformer/modeling_extractor.py","file_url":"https://github.com/huawei-noah/Pretrained-Language-Model/blob/HEAD/AutoTinyBERT/transformer/modeling_extractor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f868beb1186a0be6"}},{"code_sha256_prefix":"a788435baa6f3652","entry":"SuperBertLayerNorm","repo":"huawei-noah/Pretrained-Language-Model","repo_kind":"found_in_text","path":"AutoTinyBERT/transformer/modeling_extractor.py","file_url":"https://github.com/huawei-noah/Pretrained-Language-Model/blob/HEAD/AutoTinyBERT/transformer/modeling_extractor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a788435baa6f3652"}},{"code_sha256_prefix":"42db1dab8b861073","entry":"SuperBertOutput","repo":"huawei-noah/Pretrained-Language-Model","repo_kind":"found_in_text","path":"AutoTinyBERT/transformer/modeling_extractor.py","file_url":"https://github.com/huawei-noah/Pretrained-Language-Model/blob/HEAD/AutoTinyBERT/transformer/modeling_extractor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"42db1dab8b861073"}},{"code_sha256_prefix":"85b2789fba54e4c8","entry":"SuperBertSelfAttention","repo":"huawei-noah/Pretrained-Language-Model","repo_kind":"found_in_text","path":"AutoTinyBERT/transformer/modeling_extractor.py","file_url":"https://github.com/huawei-noah/Pretrained-Language-Model/blob/HEAD/AutoTinyBERT/transformer/modeling_extractor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"85b2789fba54e4c8"}},{"code_sha256_prefix":"47debaa2ae1929ad","entry":"SuperBertSelfOutput","repo":"huawei-noah/Pretrained-Language-Model","repo_kind":"found_in_text","path":"AutoTinyBERT/transformer/modeling_extractor.py","file_url":"https://github.com/huawei-noah/Pretrained-Language-Model/blob/HEAD/AutoTinyBERT/transformer/modeling_extractor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"47debaa2ae1929ad"}},{"code_sha256_prefix":"f2877934d142e015","entry":"SuperLinear","repo":"huawei-noah/Pretrained-Language-Model","repo_kind":"found_in_text","path":"AutoTinyBERT/transformer/modeling_extractor.py","file_url":"https://github.com/huawei-noah/Pretrained-Language-Model/blob/HEAD/AutoTinyBERT/transformer/modeling_extractor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f2877934d142e015"}},{"code_sha256_prefix":"8f3031bdf0cdcbce","entry":"sample_bias","repo":"huawei-noah/Pretrained-Language-Model","repo_kind":"found_in_text","path":"AutoTinyBERT/transformer/modeling_extractor.py","file_url":"https://github.com/huawei-noah/Pretrained-Language-Model/blob/HEAD/AutoTinyBERT/transformer/modeling_extractor.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8f3031bdf0cdcbce"}},{"code_sha256_prefix":"7ae41cd971c17070","entry":"sample_weight","repo":"huawei-noah/Pretrained-Language-Model","repo_kind":"found_in_text","path":"AutoTinyBERT/transformer/modeling_extractor.py","file_url":"https://github.com/huawei-noah/Pretrained-Language-Model/blob/HEAD/AutoTinyBERT/transformer/modeling_extractor.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7ae41cd971c17070"}},{"code_sha256_prefix":"4609348007b5c7f7","entry":"L0Module","repo":"jongwooko/nash-pruning-official","repo_kind":"listed","path":"models/l0_module.py","file_url":"https://github.com/jongwooko/nash-pruning-official/blob/HEAD/models/l0_module.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4609348007b5c7f7"}},{"code_sha256_prefix":"0c8206b768637d45","entry":"SuperBertAttention","repo":"huawei-noah/Pretrained-Language-Model","repo_kind":"found_in_text","path":"AutoTinyBERT/transformer/modeling_extractor.py","file_url":"https://github.com/huawei-noah/Pretrained-Language-Model/blob/HEAD/AutoTinyBERT/transformer/modeling_extractor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0c8206b768637d45"}},{"code_sha256_prefix":"48eeeaddaa79fd07","entry":"SuperBertLayer","repo":"huawei-noah/Pretrained-Language-Model","repo_kind":"found_in_text","path":"AutoTinyBERT/transformer/modeling_extractor.py","file_url":"https://github.com/huawei-noah/Pretrained-Language-Model/blob/HEAD/AutoTinyBERT/transformer/modeling_extractor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"48eeeaddaa79fd07"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}