{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2602-17687","title":"IRPAPERS: A Visual Document Benchmark for Scientific Retrieval and Question Answering","arxiv_id":"2602.17687","date":"2026-02-05","proceeding":null,"authors":["Connor Shorten","Augustas Skaburskas","Daniel M. Jones","Charles Pierse","Roberto Esposito","John Trengrove","Etienne Dilocker","Bob van Luijt"],"abstract":"AI systems have achieved remarkable success in processing text and relational data, yet visual document processing remains relatively underexplored. Whereas traditional systems require OCR transcriptions to convert these visual documents into text and metadata, recent advances in multimodal foundation models offer retrieval and generation directly from document images. This raises a key question: How do image-based systems compare to established text-based methods? We introduce IRPAPERS, a benchmark of 3,230 pages from 166 scientific papers, with both an image and an OCR transcription for each page. Using 180 needle-in-the-haystack questions, we compare image- and text-based retrieval and question answering systems. Text retrieval using Arctic 2.0 embeddings, BM25, and hybrid text search achieved 46% Recall@1, 78% Recall@5, and 91% Recall@20, while image-based retrieval reaches 43%, 78%, and 93%, respectively. The two modalities exhibit complementary failures, enabling multimodal hybrid search to outperform either alone, achieving 49% Recall@1, 81% Recall@5, and 95% Recall@20. We further evaluate efficiency-performance tradeoffs with MUVERA and assess multiple multi-vector image embedding models. Among closed-source models, Cohere Embed v4 page image embeddings outperform Voyage 3 Large text embeddings and all tested open-source models, achieving 58% Recall@1, 87% Recall@5, and 97% Recall@20. For question answering, text-based RAG systems achieved higher ground-truth alignment than image-based systems (0.82 vs. 0.71), and both benefit substantially from increased retrieval depth, with multi-document retrieval outperforming oracle single-document retrieval. We analyze the complementary limitations of unimodal text and image representations and identify question types that require one modality over the other. The IRPAPERS dataset and all experimental code are publicly available.","url_abs":"https://arxiv.org/abs/2602.17687","url_pdf":"https://arxiv.org/pdf/2602.17687","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2602.17687","atlas_url":"https://app.syntology.ai/?focus=2602.17687","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2602.17687"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/weaviate/IRPAPERS","reach":{"status":"ok","spdx":"MIT"}},{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/weaviate/query-agent-benchmarking","reach":null}],"summary":{"unverified":12},"by_repo_kind":{"found_in_text":{"samples":12,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c417197188899940","entry":"aggregate_metrics","repo":"weaviate/query-agent-benchmarking","repo_kind":"found_in_text","path":"query_agent_benchmarking/internal/core/domain/analysis.py","file_url":"https://github.com/weaviate/query-agent-benchmarking/blob/HEAD/query_agent_benchmarking/internal/core/domain/analysis.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"c417197188899940"}},{"code_sha256_prefix":"512f0dd6b6ba2b40","entry":"get_named_vector_target_entry","repo":"weaviate/query-agent-benchmarking","repo_kind":"found_in_text","path":"query_agent_benchmarking/internal/config/config.py","file_url":"https://github.com/weaviate/query-agent-benchmarking/blob/HEAD/query_agent_benchmarking/internal/config/config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"512f0dd6b6ba2b40"}},{"code_sha256_prefix":"0443ead669787717","entry":"get_object_by_dataset_id","repo":"weaviate/query-agent-benchmarking","repo_kind":"found_in_text","path":"query_agent_benchmarking/internal/core/domain/display.py","file_url":"https://github.com/weaviate/query-agent-benchmarking/blob/HEAD/query_agent_benchmarking/internal/core/domain/display.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"0443ead669787717"}},{"code_sha256_prefix":"ac61116f90ca6017","entry":"get_system_prompt","repo":"weaviate/query-agent-benchmarking","repo_kind":"found_in_text","path":"query_agent_benchmarking/internal/config/qa_system_prompt_registry.py","file_url":"https://github.com/weaviate/query-agent-benchmarking/blob/HEAD/query_agent_benchmarking/internal/config/qa_system_prompt_registry.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"ac61116f90ca6017"}},{"code_sha256_prefix":"3dbd15749a6ef77f","entry":"load_config","repo":"weaviate/query-agent-benchmarking","repo_kind":"found_in_text","path":"query_agent_benchmarking/internal/config/loader.py","file_url":"https://github.com/weaviate/query-agent-benchmarking/blob/HEAD/query_agent_benchmarking/internal/config/loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"3dbd15749a6ef77f"}},{"code_sha256_prefix":"53596aa180622122","entry":"make_json_serializable","repo":"weaviate/query-agent-benchmarking","repo_kind":"found_in_text","path":"query_agent_benchmarking/internal/core/domain/display.py","file_url":"https://github.com/weaviate/query-agent-benchmarking/blob/HEAD/query_agent_benchmarking/internal/core/domain/display.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"53596aa180622122"}},{"code_sha256_prefix":"056bbbb54bdfab38","entry":"merge_configs","repo":"weaviate/query-agent-benchmarking","repo_kind":"found_in_text","path":"query_agent_benchmarking/internal/config/loader.py","file_url":"https://github.com/weaviate/query-agent-benchmarking/blob/HEAD/query_agent_benchmarking/internal/config/loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"056bbbb54bdfab38"}},{"code_sha256_prefix":"2f8c08bb2bb41820","entry":"parse_extra_metrics","repo":"weaviate/query-agent-benchmarking","repo_kind":"found_in_text","path":"query_agent_benchmarking/internal/core/domain/metrics_config.py","file_url":"https://github.com/weaviate/query-agent-benchmarking/blob/HEAD/query_agent_benchmarking/internal/core/domain/metrics_config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"2f8c08bb2bb41820"}},{"code_sha256_prefix":"4a6c22418652756d","entry":"print_results_comparison","repo":"weaviate/query-agent-benchmarking","repo_kind":"found_in_text","path":"query_agent_benchmarking/internal/core/domain/display.py","file_url":"https://github.com/weaviate/query-agent-benchmarking/blob/HEAD/query_agent_benchmarking/internal/core/domain/display.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"4a6c22418652756d"}},{"code_sha256_prefix":"425af887cca74d94","entry":"resolve_metrics_profile","repo":"weaviate/query-agent-benchmarking","repo_kind":"found_in_text","path":"query_agent_benchmarking/internal/core/domain/metrics_config.py","file_url":"https://github.com/weaviate/query-agent-benchmarking/blob/HEAD/query_agent_benchmarking/internal/core/domain/metrics_config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"425af887cca74d94"}},{"code_sha256_prefix":"f4b44fb2924aac65","entry":"resolve_named_vector_target","repo":"weaviate/query-agent-benchmarking","repo_kind":"found_in_text","path":"query_agent_benchmarking/internal/config/config.py","file_url":"https://github.com/weaviate/query-agent-benchmarking/blob/HEAD/query_agent_benchmarking/internal/config/config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f4b44fb2924aac65"}},{"code_sha256_prefix":"9bcf4199f500223e","entry":"resolve_primary_metric","repo":"weaviate/query-agent-benchmarking","repo_kind":"found_in_text","path":"query_agent_benchmarking/internal/core/domain/metrics_config.py","file_url":"https://github.com/weaviate/query-agent-benchmarking/blob/HEAD/query_agent_benchmarking/internal/core/domain/metrics_config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"9bcf4199f500223e"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}