{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2602-00344","title":"When RAG Hurts: Diagnosing and Mitigating Attention Distraction in Retrieval-Augmented LVLMs","arxiv_id":"2602.00344","date":"2026-01-30","proceeding":"ICML","authors":["Beidi Zhao","Wenlong Deng","Xinting Liao","Yushu Li","Nazim Shaikh","Yao Nie","Xiaoxiao Li"],"abstract":"While Retrieval-Augmented Generation (RAG) is one of the dominant paradigms for enhancing Large Vision-Language Models (LVLMs) on knowledge-based VQA tasks, recent work attributes RAG failures to insufficient attention towards the retrieved context, proposing to reduce the attention allocated to image tokens. In this work, we identify a distinct failure mode that previous study overlooked: Attention Distraction (AD). When the retrieved context is sufficient (highly relevant or including the correct answer), the retrieved text suppresses the visual attention globally, and the attention on image tokens shifts away from question-relevant regions. This leads to failures on questions the model could originally answer correctly without the retrieved text. To mitigate this issue, we propose MAD-RAG, a training-free intervention that decouples visual grounding from context integration through a dual-question formulation, combined with attention mixing to preserve image-conditioned evidence. Extensive experiments on OK-VQA, E-VQA, and InfoSeek demonstrate that MAD-RAG consistently outperforms existing baselines across different model families, yielding absolute gains of up to 4.76%, 9.20%, and 6.18% over the vanilla RAG baseline. Notably, MAD-RAG rectifies up to 74.68% of failure cases with negligible computational overhead.","url_abs":"https://arxiv.org/abs/2602.00344","url_pdf":"https://arxiv.org/pdf/2602.00344","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2602.00344","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2602.00344"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/ubc-tea/MAD-RAG","reach":null}],"summary":{"unverified":4},"by_repo_kind":{"found_in_text":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"8df05f6e2d6f6a43","entry":"create_attention_wrapper","repo":"ubc-tea/MAD-RAG","repo_kind":"found_in_text","path":"Qwen25vl/run_mad_rag.py","file_url":"https://github.com/ubc-tea/MAD-RAG/blob/HEAD/Qwen25vl/run_mad_rag.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8df05f6e2d6f6a43"}},{"code_sha256_prefix":"d9ead4e2be157b0c","entry":"find_token_indices_robust","repo":"ubc-tea/MAD-RAG","repo_kind":"found_in_text","path":"Qwen25vl/run_mad_rag.py","file_url":"https://github.com/ubc-tea/MAD-RAG/blob/HEAD/Qwen25vl/run_mad_rag.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d9ead4e2be157b0c"}},{"code_sha256_prefix":"430e776aabfd0ff1","entry":"get_image_range","repo":"ubc-tea/MAD-RAG","repo_kind":"found_in_text","path":"llava15/run_mad_rag.py","file_url":"https://github.com/ubc-tea/MAD-RAG/blob/HEAD/llava15/run_mad_rag.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"430e776aabfd0ff1"}},{"code_sha256_prefix":"77e69e045e7b1369","entry":"get_image_token_range","repo":"ubc-tea/MAD-RAG","repo_kind":"found_in_text","path":"Qwen25vl/run_mad_rag.py","file_url":"https://github.com/ubc-tea/MAD-RAG/blob/HEAD/Qwen25vl/run_mad_rag.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"77e69e045e7b1369"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}