{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improving-retrieval-augmented-generation","title":"Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning","arxiv_id":"2501.15228","date":"2025-01-25","proceeding":null,"authors":["Yiqun Chen","Lingyong Yan","Weiwei Sun","Xinyu Ma","Yi Zhang","Shuaiqiang Wang","Dawei Yin","Yiming Yang","Jiaxin Mao"],"abstract":"Retrieval-augmented generation (RAG) is extensively utilized to incorporate external, current knowledge into large language models, thereby minimizing hallucinations. A standard RAG pipeline may comprise several components, such as query rewriting, document retrieval, document filtering, and answer generation. However, these components are typically optimized separately through supervised fine-tuning, which can lead to misalignments between the objectives of individual modules and the overarching aim of generating accurate answers in question-answering (QA) tasks. Although recent efforts have explored reinforcement learning (RL) to optimize specific RAG components, these approaches often focus on overly simplistic pipelines with only two components or do not adequately address the complex interdependencies and collaborative interactions among the modules. To overcome these challenges, we propose treating the RAG pipeline as a multi-agent cooperative task, with each component regarded as an RL agent. Specifically, we present MMOA-RAG, a Multi-Module joint Optimization Algorithm for RAG, which employs multi-agent reinforcement learning to harmonize all agents' goals towards a unified reward, such as the F1 score of the final answer. Experiments conducted on various QA datasets demonstrate that MMOA-RAG improves the overall pipeline performance and outperforms existing baselines. Furthermore, comprehensive ablation studies validate the contributions of individual components and the adaptability of MMOA-RAG across different RAG components and datasets. The code of MMOA-RAG is on https://github.com/chenyiqun/MMOA-RAG.","url_abs":"https://arxiv.org/abs/2501.15228v1","url_pdf":"https://arxiv.org/pdf/2501.15228v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improving-retrieval-augmented-generation","repo_url":"https://github.com/chenyiqun/mmoa-rag","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"answer-generation","task_name":"Answer Generation"},{"task_slug":"multi-agent-reinforcement-learning","task_name":"Multi-agent Reinforcement Learning"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"rag","task_name":"RAG"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"retrieval-augmented-generation","task_name":"Retrieval-augmented Generation"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bart","method_name":"BART"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"rag","method_name":"RAG"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2501.15228","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2501.15228"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chenyiqun/mmoa-rag","reach":{"status":"ok"}}],"summary":{"ran_honours":1,"ran_draft_wrong":3,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"56f425db8dfd78ff","entry":"calculate_match_ratio","repo":"chenyiqun/MMOA-RAG","repo_kind":"official","path":"LLaMA-Factory/src/llamafactory/train/ppo/trainer_qr_s_g.py","file_url":"https://github.com/chenyiqun/MMOA-RAG/blob/HEAD/LLaMA-Factory/src/llamafactory/train/ppo/trainer_qr_s_g.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"56f425db8dfd78ff"}},{"code_sha256_prefix":"3446d6ee423d9310","entry":"clean_and_split","repo":"chenyiqun/MMOA-RAG","repo_kind":"official","path":"LLaMA-Factory/src/llamafactory/train/ppo/trainer_qr_s_g.py","file_url":"https://github.com/chenyiqun/MMOA-RAG/blob/HEAD/LLaMA-Factory/src/llamafactory/train/ppo/trainer_qr_s_g.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3446d6ee423d9310"}},{"code_sha256_prefix":"fe0e2df4d9cb5edd","entry":"mean_pooling","repo":"chenyiqun/MMOA-RAG","repo_kind":"official","path":"evaluate_qr_s_g.py","file_url":"https://github.com/chenyiqun/MMOA-RAG/blob/HEAD/evaluate_qr_s_g.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe0e2df4d9cb5edd"}},{"code_sha256_prefix":"a6d95c1b612d9b81","entry":"remove_punctuation","repo":"chenyiqun/MMOA-RAG","repo_kind":"official","path":"LLaMA-Factory/src/llamafactory/train/ppo/trainer_qr_s_g.py","file_url":"https://github.com/chenyiqun/MMOA-RAG/blob/HEAD/LLaMA-Factory/src/llamafactory/train/ppo/trainer_qr_s_g.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a6d95c1b612d9b81"}},{"code_sha256_prefix":"8d8661f12921b6ce","entry":"normalize_answer_final","repo":"chenyiqun/MMOA-RAG","repo_kind":"official","path":"evaluate_qr_s_g.py","file_url":"https://github.com/chenyiqun/MMOA-RAG/blob/HEAD/evaluate_qr_s_g.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8d8661f12921b6ce"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}