{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/conpet-continual-parameter-efficient-tuning","title":"ConPET: Continual Parameter-Efficient Tuning for Large Language Models","arxiv_id":"2309.14763","date":"2023-09-26","proceeding":null,"authors":["Chenyang Song","Xu Han","Zheni Zeng","Kuai Li","Chen Chen","Zhiyuan Liu","Maosong Sun","Tao Yang"],"abstract":"Continual learning necessitates the continual adaptation of models to newly emerging tasks while minimizing the catastrophic forgetting of old ones. This is extremely challenging for large language models (LLMs) with vanilla full-parameter tuning due to high computation costs, memory consumption, and forgetting issue. Inspired by the success of parameter-efficient tuning (PET), we propose Continual Parameter-Efficient Tuning (ConPET), a generalizable paradigm for continual task adaptation of LLMs with task-number-independent training complexity. ConPET includes two versions with different application scenarios. First, Static ConPET can adapt former continual learning methods originally designed for relatively smaller models to LLMs through PET and a dynamic replay strategy, which largely reduces the tuning costs and alleviates the over-fitting and forgetting issue. Furthermore, to maintain scalability, Dynamic ConPET adopts separate PET modules for different tasks and a PET module selector for dynamic optimal selection. In our extensive experiments, the adaptation of Static ConPET helps multiple former methods reduce the scale of tunable parameters by over 3,000 times and surpass the PET-only baseline by at least 5 points on five smaller benchmarks, while Dynamic ConPET gains its advantage on the largest dataset. The codes and datasets are available at https://github.com/Raincleared-Song/ConPET.","url_abs":"https://arxiv.org/abs/2309.14763v1","url_pdf":"https://arxiv.org/pdf/2309.14763v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"conpet-continual-parameter-efficient-tuning","repo_url":"https://github.com/raincleared-song/conpet","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"continual-learning","task_name":"Continual Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.14763","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.14763"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/raincleared-song/conpet","reach":{"status":"ok"}}],"summary":{"ran":5,"ran_fixture":1,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"ebe3644b3b26e77d","entry":"adapter_lora_state_dict","repo":"raincleared-song/conpet","repo_kind":"official","path":"models/adapter.py","file_url":"https://github.com/raincleared-song/conpet/blob/HEAD/models/adapter.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ebe3644b3b26e77d"}},{"code_sha256_prefix":"9b4dff79d5e6102c","entry":"apply_rotary_pos_emb","repo":"raincleared-song/conpet","repo_kind":"official","path":"bmt_models/llama_lora.py","file_url":"https://github.com/raincleared-song/conpet/blob/HEAD/bmt_models/llama_lora.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b4dff79d5e6102c"}},{"code_sha256_prefix":"6a1a4730f0f39719","entry":"get_batch_limit","repo":"raincleared-song/conpet","repo_kind":"official","path":"global_var.py","file_url":"https://github.com/raincleared-song/conpet/blob/HEAD/global_var.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6a1a4730f0f39719"}},{"code_sha256_prefix":"5e1b4bd03e3092f5","entry":"get_batch_size_map","repo":"raincleared-song/conpet","repo_kind":"official","path":"global_var.py","file_url":"https://github.com/raincleared-song/conpet/blob/HEAD/global_var.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5e1b4bd03e3092f5"}},{"code_sha256_prefix":"8a4c4ad38ecee0fe","entry":"get_epoch_map","repo":"raincleared-song/conpet","repo_kind":"official","path":"global_var.py","file_url":"https://github.com/raincleared-song/conpet/blob/HEAD/global_var.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8a4c4ad38ecee0fe"}},{"code_sha256_prefix":"8b48cf849a2af11c","entry":"load_vocab","repo":"raincleared-song/conpet","repo_kind":"official","path":"bmt_models/bee_tokenizer.py","file_url":"https://github.com/raincleared-song/conpet/blob/HEAD/bmt_models/bee_tokenizer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8b48cf849a2af11c"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"raincleared-song/conpet","repo_kind":"official","path":"bmt_models/llama_lora.py","file_url":"https://github.com/raincleared-song/conpet/blob/HEAD/bmt_models/llama_lora.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"cd99ddf3ebf977c5","entry":"load_tf_weights_in_bert","repo":"raincleared-song/conpet","repo_kind":"official","path":"models/bert_lora.py","file_url":"https://github.com/raincleared-song/conpet/blob/HEAD/models/bert_lora.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cd99ddf3ebf977c5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}