{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/seekr-selective-attention-guided-knowledge","title":"SEEKR: Selective Attention-Guided Knowledge Retention for Continual Learning of Large Language Models","arxiv_id":"2411.06171","date":"2024-11-09","proceeding":null,"authors":["Jinghan He","Haiyun Guo","Kuan Zhu","Zihan Zhao","Ming Tang","Jinqiao Wang"],"abstract":"Continual learning (CL) is crucial for language models to dynamically adapt to the evolving real-world demands. To mitigate the catastrophic forgetting problem in CL, data replay has been proven a simple and effective strategy, and the subsequent data-replay-based distillation can further enhance the performance. However, existing methods fail to fully exploit the knowledge embedded in models from previous tasks, resulting in the need for a relatively large number of replay samples to achieve good results. In this work, we first explore and emphasize the importance of attention weights in knowledge retention, and then propose a SElective attEntion-guided Knowledge Retention method (SEEKR) for data-efficient replay-based continual learning of large language models (LLMs). Specifically, SEEKR performs attention distillation on the selected attention heads for finer-grained knowledge retention, where the proposed forgettability-based and task-sensitivity-based measures are used to identify the most valuable attention heads. Experimental results on two continual learning benchmarks for LLMs demonstrate the superiority of SEEKR over the existing methods on both performance and efficiency. Explicitly, SEEKR achieves comparable or even better performance with only 1/10 of the replayed data used by other methods, and reduces the proportion of replayed data to 1%.","url_abs":"https://arxiv.org/abs/2411.06171v1","url_pdf":"https://arxiv.org/pdf/2411.06171v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"seekr-selective-attention-guided-knowledge","repo_url":"https://github.com/jinghan1he/seekr","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"continual-learning","task_name":"Continual Learning"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2411.06171","atlas_url":"https://app.syntology.ai/?focus=2411.06171","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.06171"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jinghan1he/seekr","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/jinghan1he/SEEKR","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":8},"by_repo_kind":{"official":{"samples":8,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f79824eb3f619c8c","entry":"cal_replay_logits_distill_loss","repo":"jinghan1he/SEEKR","repo_kind":"official","path":"training/seekr.py","file_url":"https://github.com/jinghan1he/SEEKR/blob/HEAD/training/seekr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f79824eb3f619c8c"}},{"code_sha256_prefix":"d0e2c2740b89ee0a","entry":"get_round","repo":"jinghan1he/SEEKR","repo_kind":"official","path":"show_results.py","file_url":"https://github.com/jinghan1he/SEEKR/blob/HEAD/show_results.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d0e2c2740b89ee0a"}},{"code_sha256_prefix":"2fb70b89ac405695","entry":"get_start_task","repo":"jinghan1he/SEEKR","repo_kind":"official","path":"utils/model_utils.py","file_url":"https://github.com/jinghan1he/SEEKR/blob/HEAD/utils/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2fb70b89ac405695"}},{"code_sha256_prefix":"fac566b76c315c60","entry":"get_task_name","repo":"jinghan1he/SEEKR","repo_kind":"official","path":"show_results.py","file_url":"https://github.com/jinghan1he/SEEKR/blob/HEAD/show_results.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fac566b76c315c60"}},{"code_sha256_prefix":"b906535413fece94","entry":"postprocess","repo":"jinghan1he/SEEKR","repo_kind":"official","path":"evaluations/eval_Py150.py","file_url":"https://github.com/jinghan1he/SEEKR/blob/HEAD/evaluations/eval_Py150.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b906535413fece94"}},{"code_sha256_prefix":"6e4c7308dedfbfb6","entry":"read","repo":"jinghan1he/SEEKR","repo_kind":"official","path":"show_results.py","file_url":"https://github.com/jinghan1he/SEEKR/blob/HEAD/show_results.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6e4c7308dedfbfb6"}},{"code_sha256_prefix":"954ca5bdca3d81ea","entry":"resolve","repo":"jinghan1he/SEEKR","repo_kind":"official","path":"evaluations/eval_CStance.py","file_url":"https://github.com/jinghan1he/SEEKR/blob/HEAD/evaluations/eval_CStance.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"954ca5bdca3d81ea"}},{"code_sha256_prefix":"1d03f12631dce7ae","entry":"resolve","repo":"jinghan1he/SEEKR","repo_kind":"official","path":"evaluations/eval_ScienceQA.py","file_url":"https://github.com/jinghan1he/SEEKR/blob/HEAD/evaluations/eval_ScienceQA.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1d03f12631dce7ae"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}