{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cpet-effective-parameter-efficient-tuning-for","title":"CA-LoRA: Adapting Existing LoRA for Compressed LLMs to Enable Efficient Multi-Tasking on Personal Devices","arxiv_id":"2307.07705","date":"2023-07-15","proceeding":null,"authors":["Weilin Zhao","Yuxiang Huang","Xu Han","Zhiyuan Liu","Zhengyan Zhang","Kuai Li","Chen Chen","Tao Yang","Maosong Sun"],"abstract":"Recently, there has been a demand to deploy Large Language Models (LLMs) on personal devices such as laptops and smartphones. These LLMs have different model variants when handling different tasks. However, personal devices have limited resources and require reduced storage overhead. To address this, there are two key methods available: the first is model compression, which compresses LLMs into smaller sizes; the second is LoRA, which can transfer an LLM to other tasks with very few parameters, avoiding the storage of multiple model variants in multi-task scenarios by only preserving LoRAs. However, our experiments show that directly combining these two methods yields sub-optimal performance. Considering that the open-source community has already contributed many LoRAs to LLMs, we propose to adapt these existing LoRAs from the LLMs to their compressed version and introduce a Compression-Aware LoRA (CA-LoRA) framework. We incorporate knowledge inheritance and recovery strategies to recover the lost knowledge caused by model compression. Experiment results demonstrate that CA-LoRA outperforms the vanilla LoRA methods applied to a compressed LLM and achieves comparable performance to the non-compressed LLM with existing LoRA modules. The source code of CA-LoRA is available at https://github.com/thunlp/CA-LoRA.","url_abs":"https://arxiv.org/abs/2307.07705v3","url_pdf":"https://arxiv.org/pdf/2307.07705v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cpet-effective-parameter-efficient-tuning-for","repo_url":"https://github.com/thunlp/ca-lora","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}}],"tasks":[{"task_slug":"model-compression","task_name":"Model Compression"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2307.07705","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.07705"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thunlp/ca-lora","reach":{"status":"ok"}}],"summary":{"ran":5,"unverified":2},"by_repo_kind":{"official":{"samples":7,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"c8e969fe51c6f843","entry":"add_model_config_args","repo":"thunlp/ca-lora","repo_kind":"official","path":"src/section-4.1/model_center/arguments.py","file_url":"https://github.com/thunlp/ca-lora/blob/HEAD/src/section-4.1/model_center/arguments.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c8e969fe51c6f843"}},{"code_sha256_prefix":"6c611d5f4b45ff52","entry":"add_training_args","repo":"thunlp/ca-lora","repo_kind":"official","path":"src/section-4.1/model_center/arguments.py","file_url":"https://github.com/thunlp/ca-lora/blob/HEAD/src/section-4.1/model_center/arguments.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6c611d5f4b45ff52"}},{"code_sha256_prefix":"d4004fed05d494ec","entry":"gen_prompt","repo":"thunlp/ca-lora","repo_kind":"official","path":"src/section-4.2.1/instruct-eval/bbh.py","file_url":"https://github.com/thunlp/ca-lora/blob/HEAD/src/section-4.2.1/instruct-eval/bbh.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d4004fed05d494ec"}},{"code_sha256_prefix":"f85537d953a50f7a","entry":"gen_prompt","repo":"thunlp/ca-lora","repo_kind":"official","path":"src/section-4.2.1/instruct-eval/crass.py","file_url":"https://github.com/thunlp/ca-lora/blob/HEAD/src/section-4.2.1/instruct-eval/crass.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f85537d953a50f7a"}},{"code_sha256_prefix":"20d15978d45a2320","entry":"gen_prompt","repo":"thunlp/ca-lora","repo_kind":"official","path":"src/section-4.2.1/instruct-eval/drop.py","file_url":"https://github.com/thunlp/ca-lora/blob/HEAD/src/section-4.2.1/instruct-eval/drop.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"20d15978d45a2320"}},{"code_sha256_prefix":"8fb78069f225a973","entry":"filter_dataset","repo":"thunlp/ca-lora","repo_kind":"official","path":"src/section-4.2.1/instruct-eval/drop.py","file_url":"https://github.com/thunlp/ca-lora/blob/HEAD/src/section-4.2.1/instruct-eval/drop.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8fb78069f225a973"}},{"code_sha256_prefix":"8884c5286ed038b8","entry":"load_data","repo":"thunlp/ca-lora","repo_kind":"official","path":"src/section-4.2.1/instruct-eval/hhh.py","file_url":"https://github.com/thunlp/ca-lora/blob/HEAD/src/section-4.2.1/instruct-eval/hhh.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8884c5286ed038b8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}