{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vidorag-visual-document-retrieval-augmented","title":"ViDoRAG: Visual Document Retrieval-Augmented Generation via Dynamic Iterative Reasoning Agents","arxiv_id":"2502.18017","date":"2025-02-25","proceeding":null,"authors":["Qiuchen Wang","Ruixue Ding","Zehui Chen","Weiqi Wu","Shihang Wang","Pengjun Xie","Feng Zhao"],"abstract":"Understanding information from visually rich documents remains a significant challenge for traditional Retrieval-Augmented Generation (RAG) methods. Existing benchmarks predominantly focus on image-based question answering (QA), overlooking the fundamental challenges of efficient retrieval, comprehension, and reasoning within dense visual documents. To bridge this gap, we introduce ViDoSeek, a novel dataset designed to evaluate RAG performance on visually rich documents requiring complex reasoning. Based on it, we identify key limitations in current RAG approaches: (i) purely visual retrieval methods struggle to effectively integrate both textual and visual features, and (ii) previous approaches often allocate insufficient reasoning tokens, limiting their effectiveness. To address these challenges, we propose ViDoRAG, a novel multi-agent RAG framework tailored for complex reasoning across visual documents. ViDoRAG employs a Gaussian Mixture Model (GMM)-based hybrid strategy to effectively handle multi-modal retrieval. To further elicit the model's reasoning capabilities, we introduce an iterative agent workflow incorporating exploration, summarization, and reflection, providing a framework for investigating test-time scaling in RAG domains. Extensive experiments on ViDoSeek validate the effectiveness and generalization of our approach. Notably, ViDoRAG outperforms existing methods by over 10% on the competitive ViDoSeek benchmark.","url_abs":"https://arxiv.org/abs/2502.18017v1","url_pdf":"https://arxiv.org/pdf/2502.18017v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vidorag-visual-document-retrieval-augmented","repo_url":"https://github.com/Alibaba-NLP/ViDoRAG","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"rag","task_name":"RAG"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"retrieval-augmented-generation","task_name":"Retrieval-augmented Generation"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bart","method_name":"BART"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"rag","method_name":"RAG"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.18017","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.18017"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Alibaba-NLP/ViDoRAG","reach":null}],"summary":{"ran_draft_wrong":1,"unverified":5},"by_repo_kind":{"official":{"samples":6,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"34692b533654cd8b","entry":"extract_json","repo":"Alibaba-NLP/ViDoRAG","repo_kind":"official","path":"vidorag_agents.py","file_url":"https://github.com/Alibaba-NLP/ViDoRAG/blob/HEAD/vidorag_agents.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"34692b533654cd8b"}},{"code_sha256_prefix":"83fd2d0e53cc03d0","entry":"Inspector","repo":"Alibaba-NLP/ViDoRAG","repo_kind":"official","path":"vidorag_agents.py","file_url":"https://github.com/Alibaba-NLP/ViDoRAG/blob/HEAD/vidorag_agents.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"83fd2d0e53cc03d0"}},{"code_sha256_prefix":"a8a4b02cf71aebf6","entry":"Seeker","repo":"Alibaba-NLP/ViDoRAG","repo_kind":"official","path":"vidorag_agents.py","file_url":"https://github.com/Alibaba-NLP/ViDoRAG/blob/HEAD/vidorag_agents.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a8a4b02cf71aebf6"}},{"code_sha256_prefix":"65c25bf6cf01ae91","entry":"Synthesizer","repo":"Alibaba-NLP/ViDoRAG","repo_kind":"official","path":"vidorag_agents.py","file_url":"https://github.com/Alibaba-NLP/ViDoRAG/blob/HEAD/vidorag_agents.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"65c25bf6cf01ae91"}},{"code_sha256_prefix":"2941125b6c72c7f2","entry":"ViDoRAG_Agents","repo":"Alibaba-NLP/ViDoRAG","repo_kind":"official","path":"vidorag_agents.py","file_url":"https://github.com/Alibaba-NLP/ViDoRAG/blob/HEAD/vidorag_agents.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2941125b6c72c7f2"}},{"code_sha256_prefix":"360050312d8ad6ea","entry":"concat_images_with_bbox","repo":"Alibaba-NLP/ViDoRAG","repo_kind":"official","path":"vidorag_agents.py","file_url":"https://github.com/Alibaba-NLP/ViDoRAG/blob/HEAD/vidorag_agents.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"360050312d8ad6ea"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}