{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mitigating-catastrophic-forgetting-in-large","title":"Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal","arxiv_id":"2403.01244","date":"2024-03-02","proceeding":null,"authors":["Jianheng Huang","Leyang Cui","Ante Wang","Chengyi Yang","Xinting Liao","Linfeng Song","Junfeng Yao","Jinsong Su"],"abstract":"Large language models (LLMs) suffer from catastrophic forgetting during continual learning. Conventional rehearsal-based methods rely on previous training data to retain the model's ability, which may not be feasible in real-world applications. When conducting continual learning based on a publicly-released LLM checkpoint, the availability of the original training data may be non-existent. To address this challenge, we propose a framework called Self-Synthesized Rehearsal (SSR) that uses the LLM to generate synthetic instances for rehearsal. Concretely, we first employ the base LLM for in-context learning to generate synthetic instances. Subsequently, we utilize the latest LLM to refine the instance outputs based on the synthetic inputs, preserving its acquired ability. Finally, we select diverse high-quality synthetic instances for rehearsal in future stages. Experimental results demonstrate that SSR achieves superior or comparable performance compared to conventional rehearsal-based approaches while being more data-efficient. Besides, SSR effectively preserves the generalization capabilities of LLMs in general domains.","url_abs":"https://arxiv.org/abs/2403.01244v2","url_pdf":"https://arxiv.org/pdf/2403.01244v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mitigating-catastrophic-forgetting-in-large","repo_url":"https://github.com/deeplearnxmu/ssr","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"continual-learning","task_name":"Continual Learning"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"}],"methods":[{"method_slug":"base","method_name":"BASE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2403.01244","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.01244"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deeplearnxmu/ssr","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/DeepLearnXMU/SSR","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":1,"ran_draft_wrong":2,"ran":3,"unverified":6},"by_repo_kind":{"official":{"samples":12,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"19f01570b0a18e2b","entry":"softmax","repo":"DeepLearnXMU/SSR","repo_kind":"official","path":"mmlu_test/evaluate.py","file_url":"https://github.com/DeepLearnXMU/SSR/blob/HEAD/mmlu_test/evaluate.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"19f01570b0a18e2b"}},{"code_sha256_prefix":"cd763eaf1ac287e7","entry":"format_example","repo":"DeepLearnXMU/SSR","repo_kind":"official","path":"mmlu_test/evaluate.py","file_url":"https://github.com/DeepLearnXMU/SSR/blob/HEAD/mmlu_test/evaluate.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cd763eaf1ac287e7"}},{"code_sha256_prefix":"6ab745408cb8648b","entry":"format_subject","repo":"DeepLearnXMU/SSR","repo_kind":"official","path":"mmlu_test/evaluate.py","file_url":"https://github.com/DeepLearnXMU/SSR/blob/HEAD/mmlu_test/evaluate.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6ab745408cb8648b"}},{"code_sha256_prefix":"3383488ea82b6c15","entry":"gen_prompt","repo":"DeepLearnXMU/SSR","repo_kind":"official","path":"mmlu_test/evaluate_flan.py","file_url":"https://github.com/DeepLearnXMU/SSR/blob/HEAD/mmlu_test/evaluate_flan.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3383488ea82b6c15"}},{"code_sha256_prefix":"37d029794d683c68","entry":"gen_prompt","repo":"DeepLearnXMU/SSR","repo_kind":"official","path":"mmlu_test/evaluate_causal.py","file_url":"https://github.com/DeepLearnXMU/SSR/blob/HEAD/mmlu_test/evaluate_causal.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"37d029794d683c68"}},{"code_sha256_prefix":"8fde668203acbef4","entry":"times","repo":"DeepLearnXMU/SSR","repo_kind":"official","path":"custom/icl_gen/complete_param_alpaca.py","file_url":"https://github.com/DeepLearnXMU/SSR/blob/HEAD/custom/icl_gen/complete_param_alpaca.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8fde668203acbef4"}},{"code_sha256_prefix":"432838d3c5f2ab75","entry":"get_adaptive_bins","repo":"DeepLearnXMU/SSR","repo_kind":"official","path":"mmlu_test/calib_tools.py","file_url":"https://github.com/DeepLearnXMU/SSR/blob/HEAD/mmlu_test/calib_tools.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"432838d3c5f2ab75"}},{"code_sha256_prefix":"3c57d0ee2854879f","entry":"get_encoder","repo":"DeepLearnXMU/SSR","repo_kind":"official","path":"mmlu_test/crop.py","file_url":"https://github.com/DeepLearnXMU/SSR/blob/HEAD/mmlu_test/crop.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3c57d0ee2854879f"}},{"code_sha256_prefix":"4f76173dfe4b0c11","entry":"get_pairs","repo":"DeepLearnXMU/SSR","repo_kind":"official","path":"mmlu_test/crop.py","file_url":"https://github.com/DeepLearnXMU/SSR/blob/HEAD/mmlu_test/crop.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4f76173dfe4b0c11"}},{"code_sha256_prefix":"0af29c87aa4bd6a8","entry":"mean","repo":"DeepLearnXMU/SSR","repo_kind":"official","path":"mmlu_test/calib_tools.py","file_url":"https://github.com/DeepLearnXMU/SSR/blob/HEAD/mmlu_test/calib_tools.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0af29c87aa4bd6a8"}},{"code_sha256_prefix":"77933604fbca56e7","entry":"one_hot_encode","repo":"DeepLearnXMU/SSR","repo_kind":"official","path":"mmlu_test/calib_tools.py","file_url":"https://github.com/DeepLearnXMU/SSR/blob/HEAD/mmlu_test/calib_tools.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"77933604fbca56e7"}},{"code_sha256_prefix":"9f2267f6821181c7","entry":"pack_instructions","repo":"DeepLearnXMU/SSR","repo_kind":"official","path":"custom/icl_gen/complete_param_alpaca.py","file_url":"https://github.com/DeepLearnXMU/SSR/blob/HEAD/custom/icl_gen/complete_param_alpaca.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9f2267f6821181c7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}