{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/compeft-compression-for-communicating","title":"ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization","arxiv_id":"2311.13171","date":"2023-11-22","proceeding":null,"authors":["Prateek Yadav","Leshem Choshen","Colin Raffel","Mohit Bansal"],"abstract":"Parameter-efficient fine-tuning (PEFT) techniques make it possible to efficiently adapt a language model to create \"expert\" models that specialize to new tasks or domains. Recent techniques in model merging and compositional generalization leverage these expert models by dynamically composing modules to improve zero/few-shot generalization. Despite the efficiency of PEFT methods, the size of expert models can make it onerous to retrieve expert models per query over high-latency networks like the Internet or serve multiple experts on a single GPU. To address these issues, we present ComPEFT, a novel method for compressing fine-tuning residuals (task vectors) of PEFT based models. ComPEFT employs sparsification and ternary quantization to reduce the size of the PEFT module without performing any additional retraining while preserving or enhancing model performance. In extensive evaluation across T5, T0, and LLaMA-based models with 200M - 65B parameters, ComPEFT achieves compression ratios of 8x - 50x. In particular, we show that ComPEFT improves with scale - stronger models exhibit higher compressibility and better performance. For example, we show that ComPEFT applied to LLaMA outperforms QLoRA by 4.16% on MMLU with a storage size reduction of up to 26x. In addition, we show that the compressed experts produced by ComPEFT maintain few-shot compositional generalization capabilities, facilitate efficient communication and computation, and exhibit enhanced performance when merged. Lastly, we provide an analysis of different method components, compare it with other PEFT methods, and test ComPEFT's efficacy for compressing the residual of full-finetuning. Our code is available at https://github.com/prateeky2806/compeft.","url_abs":"https://arxiv.org/abs/2311.13171v1","url_pdf":"https://arxiv.org/pdf/2311.13171v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"compeft-compression-for-communicating","repo_url":"https://github.com/prateeky2806/compeft","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"mmlu","task_name":"MMLU"},{"task_slug":"quantization","task_name":"Quantization"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"}],"methods":[{"method_slug":"adafactor","method_name":"Adafactor"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"glu","method_name":"Gated Linear Unit"},{"method_slug":"inverse-square-root-schedule","method_name":"Inverse Square Root Schedule"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"sentencepiece","method_name":"SentencePiece"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"t5","method_name":"T5"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.13171","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.13171"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/prateeky2806/compeft","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"ran":7,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":10,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"860bf587f2fbf4e0","entry":"add_ptm_to_tv","repo":"prateeky2806/compeft","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/prateeky2806/compeft/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"860bf587f2fbf4e0"}},{"code_sha256_prefix":"76cdb08bb3df4d7f","entry":"browse_all_files_in_a_directory","repo":"prateeky2806/compeft","repo_kind":"official","path":"src/merge_utils.py","file_url":"https://github.com/prateeky2806/compeft/blob/HEAD/src/merge_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"76cdb08bb3df4d7f"}},{"code_sha256_prefix":"0e04f05ef437bf17","entry":"extract_number","repo":"prateeky2806/compeft","repo_kind":"official","path":"src/merge_utils.py","file_url":"https://github.com/prateeky2806/compeft/blob/HEAD/src/merge_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"0e04f05ef437bf17"}},{"code_sha256_prefix":"f82430123a91cf21","entry":"extract_unnatural_instructions_data","repo":"prateeky2806/compeft","repo_kind":"official","path":"src/qlora/utils.py","file_url":"https://github.com/prateeky2806/compeft/blob/HEAD/src/qlora/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f82430123a91cf21"}},{"code_sha256_prefix":"dd5b558a499d4685","entry":"state_dict_to_vector","repo":"prateeky2806/compeft","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/prateeky2806/compeft/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"dd5b558a499d4685"}},{"code_sha256_prefix":"1b1a016089577bfe","entry":"vector_to_state_dict","repo":"prateeky2806/compeft","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/prateeky2806/compeft/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"1b1a016089577bfe"}},{"code_sha256_prefix":"771baa0aee5d6f6b","entry":"vector_to_state_dict","repo":"prateeky2806/compeft","repo_kind":"official","path":"src/qlora/ties_utils.py","file_url":"https://github.com/prateeky2806/compeft/blob/HEAD/src/qlora/ties_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"771baa0aee5d6f6b"}},{"code_sha256_prefix":"57a08fe41768dc0b","entry":"walk_through_all_files_in_a_directory","repo":"prateeky2806/compeft","repo_kind":"official","path":"src/merge_utils.py","file_url":"https://github.com/prateeky2806/compeft/blob/HEAD/src/merge_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"57a08fe41768dc0b"}},{"code_sha256_prefix":"6746178a5f7cbcc1","entry":"compute_metrics","repo":"prateeky2806/compeft","repo_kind":"official","path":"src/bert_eval.py","file_url":"https://github.com/prateeky2806/compeft/blob/HEAD/src/bert_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"6746178a5f7cbcc1"}},{"code_sha256_prefix":"ed1d520d4fd927b3","entry":"preprocess_function","repo":"prateeky2806/compeft","repo_kind":"official","path":"src/bert_eval.py","file_url":"https://github.com/prateeky2806/compeft/blob/HEAD/src/bert_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"ed1d520d4fd927b3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}