{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/systematic-knowledge-injection-into-large","title":"Systematic Knowledge Injection into Large Language Models via Diverse Augmentation for Domain-Specific RAG","arxiv_id":"2502.08356","date":"2025-02-12","proceeding":null,"authors":["Kushagra Bhushan","Yatin Nandwani","Dinesh Khandelwal","Sonam Gupta","Gaurav Pandey","Dinesh Raghu","Sachindra Joshi"],"abstract":"Retrieval-Augmented Generation (RAG) has emerged as a prominent method for incorporating domain knowledge into Large Language Models (LLMs). While RAG enhances response relevance by incorporating retrieved domain knowledge in the context, retrieval errors can still lead to hallucinations and incorrect answers. To recover from retriever failures, domain knowledge is injected by fine-tuning the model to generate the correct response, even in the case of retrieval errors. However, we observe that without systematic knowledge augmentation, fine-tuned LLMs may memorize new information but still fail to extract relevant domain knowledge, leading to poor performance. In this work, we present a novel framework that significantly enhances the fine-tuning process by augmenting the training data in two ways -- context augmentation and knowledge paraphrasing. In context augmentation, we create multiple training samples for a given QA pair by varying the relevance of the retrieved information, teaching the model when to ignore and when to rely on retrieved content. In knowledge paraphrasing, we fine-tune with multiple answers to the same question, enabling LLMs to better internalize specialized knowledge. To mitigate catastrophic forgetting due to fine-tuning, we add a domain-specific identifier to a question and also utilize a replay buffer containing general QA pairs. Experimental results demonstrate the efficacy of our method over existing techniques, achieving up to 10\\% relative gain in token-level recall while preserving the LLM's generalization capabilities.","url_abs":"https://arxiv.org/abs/2502.08356v3","url_pdf":"https://arxiv.org/pdf/2502.08356v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"systematic-knowledge-injection-into-large","repo_url":"https://github.com/kushagrabhushan/Systematic-Knowledge-Injection","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"rag","task_name":"RAG"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"retrieval-augmented-generation","task_name":"Retrieval-augmented Generation"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bart","method_name":"BART"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"rag","method_name":"RAG"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.08356","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.08356"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kushagrabhushan/Systematic-Knowledge-Injection","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":9},"by_repo_kind":{"official":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bdbd75911ca8043c","entry":"capture_chapter_title","repo":"kushagrabhushan/Systematic-Knowledge-Injection","repo_kind":"official","path":"py_scripts/create_and_index_chunks.py","file_url":"https://github.com/kushagrabhushan/Systematic-Knowledge-Injection/blob/HEAD/py_scripts/create_and_index_chunks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bdbd75911ca8043c"}},{"code_sha256_prefix":"104b7adca353b01d","entry":"force_retriever_failure","repo":"kushagrabhushan/Systematic-Knowledge-Injection","repo_kind":"official","path":"py_scripts/corrupt_retriever.py","file_url":"https://github.com/kushagrabhushan/Systematic-Knowledge-Injection/blob/HEAD/py_scripts/corrupt_retriever.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"104b7adca353b01d"}},{"code_sha256_prefix":"c2b4bd7f6c40dcdf","entry":"get_chapter_text","repo":"kushagrabhushan/Systematic-Knowledge-Injection","repo_kind":"official","path":"py_scripts/create_and_index_chunks.py","file_url":"https://github.com/kushagrabhushan/Systematic-Knowledge-Injection/blob/HEAD/py_scripts/create_and_index_chunks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c2b4bd7f6c40dcdf"}},{"code_sha256_prefix":"675b9f9a51334c87","entry":"parse_arguments","repo":"kushagrabhushan/Systematic-Knowledge-Injection","repo_kind":"official","path":"py_scripts/assign_gold_passage_ids_to_subdocs.py","file_url":"https://github.com/kushagrabhushan/Systematic-Knowledge-Injection/blob/HEAD/py_scripts/assign_gold_passage_ids_to_subdocs.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"675b9f9a51334c87"}},{"code_sha256_prefix":"b938cda3e98319eb","entry":"parse_arguments","repo":"kushagrabhushan/Systematic-Knowledge-Injection","repo_kind":"official","path":"py_scripts/corrupt_retriever.py","file_url":"https://github.com/kushagrabhushan/Systematic-Knowledge-Injection/blob/HEAD/py_scripts/corrupt_retriever.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b938cda3e98319eb"}},{"code_sha256_prefix":"0ea467579b57a36e","entry":"parse_arguments","repo":"kushagrabhushan/Systematic-Knowledge-Injection","repo_kind":"official","path":"py_scripts/create_and_index_chunks.py","file_url":"https://github.com/kushagrabhushan/Systematic-Knowledge-Injection/blob/HEAD/py_scripts/create_and_index_chunks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0ea467579b57a36e"}},{"code_sha256_prefix":"2da24db55f2ec020","entry":"parse_arguments","repo":"kushagrabhushan/Systematic-Knowledge-Injection","repo_kind":"official","path":"py_scripts/retrieve_passages_for_synthetic_data.py","file_url":"https://github.com/kushagrabhushan/Systematic-Knowledge-Injection/blob/HEAD/py_scripts/retrieve_passages_for_synthetic_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2da24db55f2ec020"}},{"code_sha256_prefix":"e1ec4ece60036dfb","entry":"populate_cache","repo":"kushagrabhushan/Systematic-Knowledge-Injection","repo_kind":"official","path":"py_scripts/retrieve_passages_for_synthetic_data.py","file_url":"https://github.com/kushagrabhushan/Systematic-Knowledge-Injection/blob/HEAD/py_scripts/retrieve_passages_for_synthetic_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e1ec4ece60036dfb"}},{"code_sha256_prefix":"5e44661cb8ad8f95","entry":"select_retrieved_ids","repo":"kushagrabhushan/Systematic-Knowledge-Injection","repo_kind":"official","path":"py_scripts/corrupt_retriever.py","file_url":"https://github.com/kushagrabhushan/Systematic-Knowledge-Injection/blob/HEAD/py_scripts/corrupt_retriever.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5e44661cb8ad8f95"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}