{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/train-attention-meta-learning-where-to-focus","title":"Train-Attention: Meta-Learning Where to Focus in Continual Knowledge Learning","arxiv_id":"2407.16920","date":"2024-07-24","proceeding":null,"authors":["Yeongbin Seo","Dongha Lee","Jinyoung Yeo"],"abstract":"Previous studies on continual knowledge learning (CKL) in large language models (LLMs) have predominantly focused on approaches such as regularization, architectural modifications, and rehearsal techniques to mitigate catastrophic forgetting. However, these methods naively inherit the inefficiencies of standard training procedures, indiscriminately applying uniform weight across all tokens, which can lead to unnecessary parameter updates and increased forgetting. To address these shortcomings, we propose a novel CKL approach termed Train-Attention-Augmented Language Model (TAALM), which enhances learning efficiency by dynamically predicting and applying weights to tokens based on their usefulness. This method employs a meta-learning framework that optimizes token importance predictions, facilitating targeted knowledge updates and minimizing forgetting. Also, we observe that existing benchmarks do not clearly exhibit the trade-off between learning and retaining, therefore we propose a new benchmark, \\textsc{LAMA-ckl}, to address this issue. Through experiments conducted on both newly introduced and established CKL benchmarks, TAALM proves the state-of-the-art performance upon the baselines, and also shows synergistic compatibility when integrated with previous CKL approaches.","url_abs":"https://arxiv.org/abs/2407.16920v1","url_pdf":"https://arxiv.org/pdf/2407.16920v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"train-attention-meta-learning-where-to-focus","repo_url":"https://github.com/ybseo-academy/TAALM","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"meta-learning","task_name":"Meta-Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.16920","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.16920"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ybseo-academy/TAALM","reach":{"status":"ok"}}],"summary":{"ran_honours":1,"ran":9,"unverified":2},"by_repo_kind":{"official":{"samples":12,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"b8301a2a061e30a2","entry":"anneal_function","repo":"ybseo-academy/TAALM","repo_kind":"official","path":"utils/RecAdam.py","file_url":"https://github.com/ybseo-academy/TAALM/blob/HEAD/utils/RecAdam.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b8301a2a061e30a2"}},{"code_sha256_prefix":"9745751dcea1a41f","entry":"giveme_label_mask","repo":"ybseo-academy/TAALM","repo_kind":"official","path":"evaluation_run.py","file_url":"https://github.com/ybseo-academy/TAALM/blob/HEAD/evaluation_run.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9745751dcea1a41f"}},{"code_sha256_prefix":"aa580f8ceb463858","entry":"giveme_label_mask0","repo":"ybseo-academy/TAALM","repo_kind":"official","path":"ta_eval.py","file_url":"https://github.com/ybseo-academy/TAALM/blob/HEAD/ta_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa580f8ceb463858"}},{"code_sha256_prefix":"6f28ab856d890d85","entry":"giveme_label_mask_list","repo":"ybseo-academy/TAALM","repo_kind":"official","path":"evaluation_run.py","file_url":"https://github.com/ybseo-academy/TAALM/blob/HEAD/evaluation_run.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6f28ab856d890d85"}},{"code_sha256_prefix":"0be42af713b24083","entry":"giveme_label_mask_list","repo":"ybseo-academy/TAALM","repo_kind":"official","path":"ta_eval.py","file_url":"https://github.com/ybseo-academy/TAALM/blob/HEAD/ta_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0be42af713b24083"}},{"code_sha256_prefix":"2594581b57a6f409","entry":"make_conv","repo":"ybseo-academy/TAALM","repo_kind":"official","path":"ta_toolbox.py","file_url":"https://github.com/ybseo-academy/TAALM/blob/HEAD/ta_toolbox.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2594581b57a6f409"}},{"code_sha256_prefix":"c52d48f5fb3ab33d","entry":"make_real_id","repo":"ybseo-academy/TAALM","repo_kind":"official","path":"TAALM_train_run.py","file_url":"https://github.com/ybseo-academy/TAALM/blob/HEAD/TAALM_train_run.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c52d48f5fb3ab33d"}},{"code_sha256_prefix":"26e2ec638208bc5c","entry":"make_task","repo":"ybseo-academy/TAALM","repo_kind":"official","path":"evaluation_run.py","file_url":"https://github.com/ybseo-academy/TAALM/blob/HEAD/evaluation_run.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"26e2ec638208bc5c"}},{"code_sha256_prefix":"f8a3fcc48b115cac","entry":"make_task","repo":"ybseo-academy/TAALM","repo_kind":"official","path":"ta_eval.py","file_url":"https://github.com/ybseo-academy/TAALM/blob/HEAD/ta_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f8a3fcc48b115cac"}},{"code_sha256_prefix":"1e1cba6114cc9188","entry":"preprocess","repo":"ybseo-academy/TAALM","repo_kind":"official","path":"TAALM_train_run.py","file_url":"https://github.com/ybseo-academy/TAALM/blob/HEAD/TAALM_train_run.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1e1cba6114cc9188"}},{"code_sha256_prefix":"57f800133f0342bc","entry":"adamw","repo":"ybseo-academy/TAALM","repo_kind":"official","path":"utils/tool_TAALM/taalm_adamw.py","file_url":"https://github.com/ybseo-academy/TAALM/blob/HEAD/utils/tool_TAALM/taalm_adamw.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"57f800133f0342bc"}},{"code_sha256_prefix":"cb4e99dd4b50f69d","entry":"load_tf_weights_in_gpt2","repo":"ybseo-academy/TAALM","repo_kind":"official","path":"utils/kadapter/Llama2_Model_Kadapter.py","file_url":"https://github.com/ybseo-academy/TAALM/blob/HEAD/utils/kadapter/Llama2_Model_Kadapter.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb4e99dd4b50f69d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}