{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/toward-general-instruction-following","title":"Toward General Instruction-Following Alignment for Retrieval-Augmented Generation","arxiv_id":"2410.09584","date":"2024-10-12","proceeding":null,"authors":["Guanting Dong","Xiaoshuai Song","Yutao Zhu","Runqi Qiao","Zhicheng Dou","Ji-Rong Wen"],"abstract":"Following natural instructions is crucial for the effective application of Retrieval-Augmented Generation (RAG) systems. Despite recent advancements in Large Language Models (LLMs), research on assessing and improving instruction-following (IF) alignment within the RAG domain remains limited. To address this issue, we propose VIF-RAG, the first automated, scalable, and verifiable synthetic pipeline for instruction-following alignment in RAG systems. We start by manually crafting a minimal set of atomic instructions (<100) and developing combination rules to synthesize and verify complex instructions for a seed set. We then use supervised models for instruction rewriting while simultaneously generating code to automate the verification of instruction quality via a Python executor. Finally, we integrate these instructions with extensive RAG and general data samples, scaling up to a high-quality VIF-RAG-QA dataset (>100k) through automated processes. To further bridge the gap in instruction-following auto-evaluation for RAG systems, we introduce FollowRAG Benchmark, which includes approximately 3K test samples, covering 22 categories of general instruction constraints and four knowledge-intensive QA datasets. Due to its robust pipeline design, FollowRAG can seamlessly integrate with different RAG benchmarks. Using FollowRAG and eight widely-used IF and foundational abilities benchmarks for LLMs, we demonstrate that VIF-RAG markedly enhances LLM performance across a broad range of general instruction constraints while effectively leveraging its capabilities in RAG scenarios. Further analysis offers practical insights for achieving IF alignment in RAG systems. Our code and datasets are released at https://FollowRAG.github.io.","url_abs":"https://arxiv.org/abs/2410.09584v1","url_pdf":"https://arxiv.org/pdf/2410.09584v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"toward-general-instruction-following","repo_url":"https://github.com/dongguanting/FollowRAG","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"rag","task_name":"RAG"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"retrieval-augmented-generation","task_name":"Retrieval-augmented Generation"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bart","method_name":"BART"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"rag","method_name":"RAG"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"set","method_name":"SET"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.09584","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.09584"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dongguanting/FollowRAG","reach":{"status":"ok"}}],"summary":{"ran":7,"unverified":2},"by_repo_kind":{"official":{"samples":9,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"d663a34433138f3b","entry":"construct_eval_prompt","repo":"dongguanting/FollowRAG","repo_kind":"official","path":"FollowRAG/eval/rag_eval.py","file_url":"https://github.com/dongguanting/FollowRAG/blob/HEAD/FollowRAG/eval/rag_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d663a34433138f3b"}},{"code_sha256_prefix":"cdcc85ca09b00f7e","entry":"count_words","repo":"dongguanting/FollowRAG","repo_kind":"official","path":"FollowRAG/utils/instruction_following_eval/instructions_util.py","file_url":"https://github.com/dongguanting/FollowRAG/blob/HEAD/FollowRAG/utils/instruction_following_eval/instructions_util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cdcc85ca09b00f7e"}},{"code_sha256_prefix":"8ad4bfd77fd4b38e","entry":"extract_rating","repo":"dongguanting/FollowRAG","repo_kind":"official","path":"FollowRAG/eval/rag_eval.py","file_url":"https://github.com/dongguanting/FollowRAG/blob/HEAD/FollowRAG/eval/rag_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8ad4bfd77fd4b38e"}},{"code_sha256_prefix":"621bb80a81f5a607","entry":"read_json","repo":"dongguanting/FollowRAG","repo_kind":"official","path":"FollowRAG/utils/util.py","file_url":"https://github.com/dongguanting/FollowRAG/blob/HEAD/FollowRAG/utils/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"621bb80a81f5a607"}},{"code_sha256_prefix":"7bb90ae0136ac9dd","entry":"read_jsonl","repo":"dongguanting/FollowRAG","repo_kind":"official","path":"FollowRAG/utils/util.py","file_url":"https://github.com/dongguanting/FollowRAG/blob/HEAD/FollowRAG/utils/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7bb90ae0136ac9dd"}},{"code_sha256_prefix":"740114bd24430c6f","entry":"read_txt","repo":"dongguanting/FollowRAG","repo_kind":"official","path":"FollowRAG/utils/util.py","file_url":"https://github.com/dongguanting/FollowRAG/blob/HEAD/FollowRAG/utils/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"740114bd24430c6f"}},{"code_sha256_prefix":"34d346853108011e","entry":"split_into_sentences","repo":"dongguanting/FollowRAG","repo_kind":"official","path":"FollowRAG/utils/instruction_following_eval/instructions_util.py","file_url":"https://github.com/dongguanting/FollowRAG/blob/HEAD/FollowRAG/utils/instruction_following_eval/instructions_util.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"34d346853108011e"}},{"code_sha256_prefix":"30de2ad5d6678c9d","entry":"call_gpt","repo":"dongguanting/FollowRAG","repo_kind":"official","path":"FollowRAG/utils/call_llm.py","file_url":"https://github.com/dongguanting/FollowRAG/blob/HEAD/FollowRAG/utils/call_llm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"30de2ad5d6678c9d"}},{"code_sha256_prefix":"f79646ddd0a03162","entry":"count_sentences","repo":"dongguanting/FollowRAG","repo_kind":"official","path":"FollowRAG/utils/instruction_following_eval/instructions_util.py","file_url":"https://github.com/dongguanting/FollowRAG/blob/HEAD/FollowRAG/utils/instruction_following_eval/instructions_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f79646ddd0a03162"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}