{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/kala-knowledge-augmented-language-model-1","title":"KALA: Knowledge-Augmented Language Model Adaptation","arxiv_id":"2204.10555","date":"2022-04-22","proceeding":"NAACL 2022 7","authors":["Minki Kang","Jinheon Baek","Sung Ju Hwang"],"abstract":"Pre-trained language models (PLMs) have achieved remarkable success on various natural language understanding tasks. Simple fine-tuning of PLMs, on the other hand, might be suboptimal for domain-specific tasks because they cannot possibly cover knowledge from all domains. While adaptive pre-training of PLMs can help them obtain domain-specific knowledge, it requires a large training cost. Moreover, adaptive pre-training can harm the PLM's performance on the downstream task by causing catastrophic forgetting of its general knowledge. To overcome such limitations of adaptive pre-training for PLM adaption, we propose a novel domain adaption framework for PLMs coined as Knowledge-Augmented Language model Adaptation (KALA), which modulates the intermediate hidden representations of PLMs with domain knowledge, consisting of entities and their relational facts. We validate the performance of our KALA on question answering and named entity recognition tasks on multiple datasets across various domains. The results show that, despite being computationally efficient, our KALA largely outperforms adaptive pre-training. Code is available at: https://github.com/Nardien/KALA/.","url_abs":"https://arxiv.org/abs/2204.10555v2","url_pdf":"https://arxiv.org/pdf/2204.10555v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"kala-knowledge-augmented-language-model-1","repo_url":"https://github.com/nardien/kala","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"domain-adaptation","task_name":"Domain Adaptation"},{"task_slug":"general-knowledge","task_name":"General Knowledge"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"named-entity-recognition-1","task_name":"Named Entity Recognition"},{"task_slug":"named-entity-recognition-ner","task_name":"Named Entity Recognition (NER)"},{"task_slug":"natural-language-understanding","task_name":"Natural Language Understanding"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"model","task_name":"model"},{"task_slug":"named-entity-recognition","task_name":"named-entity-recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2204.10555","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2204.10555"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Nardien/KALA","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/rusty1s/pytorch_scatter","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":16},"by_repo_kind":{"found_in_text":{"samples":16,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b3c87119122db68e","entry":"bold","repo":"rusty1s/pytorch_scatter","repo_kind":"found_in_text","path":"benchmark/scatter_segment.py","file_url":"https://github.com/rusty1s/pytorch_scatter/blob/HEAD/benchmark/scatter_segment.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b3c87119122db68e"}},{"code_sha256_prefix":"5dd38da29766376b","entry":"broadcast","repo":"rusty1s/pytorch_scatter","repo_kind":"found_in_text","path":"torch_scatter/utils.py","file_url":"https://github.com/rusty1s/pytorch_scatter/blob/HEAD/torch_scatter/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5dd38da29766376b"}},{"code_sha256_prefix":"865555a5b2967160","entry":"scatter_add","repo":"rusty1s/pytorch_scatter","repo_kind":"found_in_text","path":"torch_scatter/scatter.py","file_url":"https://github.com/rusty1s/pytorch_scatter/blob/HEAD/torch_scatter/scatter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"865555a5b2967160"}},{"code_sha256_prefix":"fb9b382e50fdaaea","entry":"scatter_arg_placeholder","repo":"rusty1s/pytorch_scatter","repo_kind":"found_in_text","path":"torch_scatter/placeholder.py","file_url":"https://github.com/rusty1s/pytorch_scatter/blob/HEAD/torch_scatter/placeholder.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fb9b382e50fdaaea"}},{"code_sha256_prefix":"669f6da105b0c7e0","entry":"scatter_mul","repo":"rusty1s/pytorch_scatter","repo_kind":"found_in_text","path":"torch_scatter/scatter.py","file_url":"https://github.com/rusty1s/pytorch_scatter/blob/HEAD/torch_scatter/scatter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"669f6da105b0c7e0"}},{"code_sha256_prefix":"9a4c77eb3ae0b358","entry":"scatter_placeholder","repo":"rusty1s/pytorch_scatter","repo_kind":"found_in_text","path":"torch_scatter/placeholder.py","file_url":"https://github.com/rusty1s/pytorch_scatter/blob/HEAD/torch_scatter/placeholder.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a4c77eb3ae0b358"}},{"code_sha256_prefix":"5a1b967aebe67ca2","entry":"scatter_sum","repo":"rusty1s/pytorch_scatter","repo_kind":"found_in_text","path":"torch_scatter/scatter.py","file_url":"https://github.com/rusty1s/pytorch_scatter/blob/HEAD/torch_scatter/scatter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a1b967aebe67ca2"}},{"code_sha256_prefix":"62682a1d5299574b","entry":"segment_add_coo","repo":"rusty1s/pytorch_scatter","repo_kind":"found_in_text","path":"torch_scatter/segment_coo.py","file_url":"https://github.com/rusty1s/pytorch_scatter/blob/HEAD/torch_scatter/segment_coo.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"62682a1d5299574b"}},{"code_sha256_prefix":"2fd5fa44ec7ca893","entry":"segment_add_csr","repo":"rusty1s/pytorch_scatter","repo_kind":"found_in_text","path":"torch_scatter/segment_csr.py","file_url":"https://github.com/rusty1s/pytorch_scatter/blob/HEAD/torch_scatter/segment_csr.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2fd5fa44ec7ca893"}},{"code_sha256_prefix":"7b606ef61cca7d85","entry":"segment_csr_placeholder","repo":"rusty1s/pytorch_scatter","repo_kind":"found_in_text","path":"torch_scatter/placeholder.py","file_url":"https://github.com/rusty1s/pytorch_scatter/blob/HEAD/torch_scatter/placeholder.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b606ef61cca7d85"}},{"code_sha256_prefix":"92620b8f31917880","entry":"segment_mean_coo","repo":"rusty1s/pytorch_scatter","repo_kind":"found_in_text","path":"torch_scatter/segment_coo.py","file_url":"https://github.com/rusty1s/pytorch_scatter/blob/HEAD/torch_scatter/segment_coo.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"92620b8f31917880"}},{"code_sha256_prefix":"b1bae3eea035dede","entry":"segment_mean_csr","repo":"rusty1s/pytorch_scatter","repo_kind":"found_in_text","path":"torch_scatter/segment_csr.py","file_url":"https://github.com/rusty1s/pytorch_scatter/blob/HEAD/torch_scatter/segment_csr.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1bae3eea035dede"}},{"code_sha256_prefix":"ae435f7069b6353d","entry":"segment_sum_coo","repo":"rusty1s/pytorch_scatter","repo_kind":"found_in_text","path":"torch_scatter/segment_coo.py","file_url":"https://github.com/rusty1s/pytorch_scatter/blob/HEAD/torch_scatter/segment_coo.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ae435f7069b6353d"}},{"code_sha256_prefix":"698fcf43ba296493","entry":"segment_sum_csr","repo":"rusty1s/pytorch_scatter","repo_kind":"found_in_text","path":"torch_scatter/segment_csr.py","file_url":"https://github.com/rusty1s/pytorch_scatter/blob/HEAD/torch_scatter/segment_csr.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"698fcf43ba296493"}},{"code_sha256_prefix":"5299d175b3a523b3","entry":"tensor","repo":"rusty1s/pytorch_scatter","repo_kind":"found_in_text","path":"torch_scatter/testing.py","file_url":"https://github.com/rusty1s/pytorch_scatter/blob/HEAD/torch_scatter/testing.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5299d175b3a523b3"}},{"code_sha256_prefix":"861d3ec00dfdf4aa","entry":"time_func","repo":"rusty1s/pytorch_scatter","repo_kind":"found_in_text","path":"benchmark/scatter_segment.py","file_url":"https://github.com/rusty1s/pytorch_scatter/blob/HEAD/benchmark/scatter_segment.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"861d3ec00dfdf4aa"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}