{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/continual-learning-of-large-language-models-a","title":"Continual Learning of Large Language Models: A Comprehensive Survey","arxiv_id":"2404.16789","date":"2024-04-25","proceeding":null,"authors":["Haizhou Shi","Zihao Xu","Hengyi Wang","Weiyi Qin","Wenyuan Wang","Yibin Wang","Zifeng Wang","Sayna Ebrahimi","Hao Wang"],"abstract":"The recent success of large language models (LLMs) trained on static, pre-collected, general datasets has sparked numerous research directions and applications. One such direction addresses the non-trivial challenge of integrating pre-trained LLMs into dynamic data distributions, task structures, and user preferences. Pre-trained LLMs, when tailored for specific needs, often experience significant performance degradation in previous knowledge domains -- a phenomenon known as \"catastrophic forgetting\". While extensively studied in the continual learning (CL) community, it presents new manifestations in the realm of LLMs. In this survey, we provide a comprehensive overview of the current research progress on LLMs within the context of CL. This survey is structured into four main sections: we first describe an overview of continually learning LLMs, consisting of two directions of continuity: vertical continuity (or vertical continual learning), i.e., continual adaptation from general to specific capabilities, and horizontal continuity (or horizontal continual learning), i.e., continual adaptation across time and domains (Section 3). We then summarize three stages of learning LLMs in the context of modern CL: Continual Pre-Training (CPT), Domain-Adaptive Pre-training (DAP), and Continual Fine-Tuning (CFT) (Section 4). Then we provide an overview of evaluation protocols for continual learning with LLMs, along with the current available data sources (Section 5). Finally, we discuss intriguing questions pertaining to continual learning for LLMs (Section 6). The full list of papers examined in this survey is available at https://github.com/Wang-ML-Lab/llm-continual-learning-survey.","url_abs":"https://arxiv.org/abs/2404.16789v3","url_pdf":"https://arxiv.org/pdf/2404.16789v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"continual-learning-of-large-language-models-a","repo_url":"https://github.com/wang-ml-lab/llm-continual-learning-survey","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"continual-learning-of-large-language-models-a","repo_url":"https://github.com/beyonderxx/trace","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"continual-learning","task_name":"Continual Learning"},{"task_slug":"survey","task_name":"Survey"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2404.16789","atlas_url":"https://app.syntology.ai/?focus=2404.16789","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.16789"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wang-ml-lab/llm-continual-learning-survey","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/beyonderxx/trace","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ab795c060d39c540","entry":"bleu_score","repo":"beyonderxx/trace","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/beyonderxx/trace/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ab795c060d39c540"}},{"code_sha256_prefix":"3fb7352452396dd1","entry":"caculate_bleu","repo":"beyonderxx/trace","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/beyonderxx/trace/blob/HEAD/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3fb7352452396dd1"}},{"code_sha256_prefix":"90561b6f407e4842","entry":"resolve","repo":"beyonderxx/trace","repo_kind":"official","path":"evaluations/eval_PapyrusF.py","file_url":"https://github.com/beyonderxx/trace/blob/HEAD/evaluations/eval_PapyrusF.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"90561b6f407e4842"}},{"code_sha256_prefix":"e605721addd552ac","entry":"tokenize","repo":"beyonderxx/trace","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/beyonderxx/trace/blob/HEAD/metrics.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e605721addd552ac"}},{"code_sha256_prefix":"b906535413fece94","entry":"postprocess","repo":"beyonderxx/trace","repo_kind":"official","path":"evaluations/eval_Py150.py","file_url":"https://github.com/beyonderxx/trace/blob/HEAD/evaluations/eval_Py150.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b906535413fece94"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}