{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2604-05818","title":"WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering","arxiv_id":"2604.05818","date":"2026-04-07","proceeding":null,"authors":["Yingjian Zhu","Xinming Wang","Kun Ding","Ying Wang","Bin Fan","Shiming Xiang"],"abstract":"Multi-modal Retrieval-Augmented Generation (RAG) has emerged as a highly effective paradigm for Knowledge-Based Visual Question Answering (KB-VQA). Despite recent advancements, prevailing methods still primarily depend on images as the retrieval key, and often overlook or misplace the role of Vision-Language Models (VLMs), thereby failing to leverage their potential fully. In this paper, we introduce WikiSeeker, a novel multi-modal RAG framework that bridges these gaps by proposing a multi-modal retriever and redefining the role of VLMs. Rather than serving merely as answer generators, we assign VLMs two specialized agents: a Refiner and an Inspector. The Refiner utilizes the capability of VLMs to rewrite the textual query according to the input image, significantly improving the performance of the multimodal retriever. The Inspector facilitates a decoupled generation strategy by selectively routing reliable retrieved context to another LLM for answer generation, while relying on the VLM's internal knowledge when retrieval is unreliable. Extensive experiments on EVQA, InfoSeek, and M2KR demonstrate that WikiSeeker achieves state-of-the-art performance, with substantial improvements in both retrieval accuracy and answer quality. Our code will be released on https://github.com/zhuyjan/WikiSeeker.","url_abs":"https://arxiv.org/abs/2604.05818","url_pdf":"https://arxiv.org/pdf/2604.05818","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2604.05818","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2604.05818"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/zhuyjan/WikiSeeker","reach":null}],"summary":{"ran":1,"unverified":15},"by_repo_kind":{"found_in_text":{"samples":16,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8896f4e2bafe8abc","entry":"generate_config_from_args","repo":"zhuyjan/WikiSeeker","repo_kind":"found_in_text","path":"verl/model_merger/base_model_merger.py","file_url":"https://github.com/zhuyjan/WikiSeeker/blob/HEAD/verl/model_merger/base_model_merger.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8896f4e2bafe8abc"}},{"code_sha256_prefix":"2f8815aeb8fd2a80","entry":"encode_image_to_base64","repo":"zhuyjan/WikiSeeker","repo_kind":"found_in_text","path":"gen_refined_queries.py","file_url":"https://github.com/zhuyjan/WikiSeeker/blob/HEAD/gen_refined_queries.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2f8815aeb8fd2a80"}},{"code_sha256_prefix":"6d3965d7fe3b7d5d","entry":"eval_recall","repo":"zhuyjan/WikiSeeker","repo_kind":"found_in_text","path":"run_retrieval_with_service.py","file_url":"https://github.com/zhuyjan/WikiSeeker/blob/HEAD/run_retrieval_with_service.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6d3965d7fe3b7d5d"}},{"code_sha256_prefix":"7055aa97f2bde50e","entry":"exact_match_score","repo":"zhuyjan/WikiSeeker","repo_kind":"found_in_text","path":"utils/infoseek_evaluation_utils.py","file_url":"https://github.com/zhuyjan/WikiSeeker/blob/HEAD/utils/infoseek_evaluation_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7055aa97f2bde50e"}},{"code_sha256_prefix":"5628f356ff1a3709","entry":"extract_query_from_response","repo":"zhuyjan/WikiSeeker","repo_kind":"found_in_text","path":"gen_refined_queries.py","file_url":"https://github.com/zhuyjan/WikiSeeker/blob/HEAD/gen_refined_queries.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5628f356ff1a3709"}},{"code_sha256_prefix":"d862607dc4bbee23","entry":"get_image","repo":"zhuyjan/WikiSeeker","repo_kind":"found_in_text","path":"utils/utils.py","file_url":"https://github.com/zhuyjan/WikiSeeker/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d862607dc4bbee23"}},{"code_sha256_prefix":"5ef488471c45d4b8","entry":"get_retriever","repo":"zhuyjan/WikiSeeker","repo_kind":"found_in_text","path":"utils/retriever_manager.py","file_url":"https://github.com/zhuyjan/WikiSeeker/blob/HEAD/utils/retriever_manager.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5ef488471c45d4b8"}},{"code_sha256_prefix":"259527389313b6f0","entry":"get_train_image","repo":"zhuyjan/WikiSeeker","repo_kind":"found_in_text","path":"utils/utils.py","file_url":"https://github.com/zhuyjan/WikiSeeker/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"259527389313b6f0"}},{"code_sha256_prefix":"5d45d4d98e2da070","entry":"load_csv_data","repo":"zhuyjan/WikiSeeker","repo_kind":"found_in_text","path":"utils/utils.py","file_url":"https://github.com/zhuyjan/WikiSeeker/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5d45d4d98e2da070"}},{"code_sha256_prefix":"93de3112886f0ae2","entry":"make_prompt","repo":"zhuyjan/WikiSeeker","repo_kind":"found_in_text","path":"gen_refined_queries.py","file_url":"https://github.com/zhuyjan/WikiSeeker/blob/HEAD/gen_refined_queries.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"93de3112886f0ae2"}},{"code_sha256_prefix":"921382b550348607","entry":"metric_max_over_ground_truths","repo":"zhuyjan/WikiSeeker","repo_kind":"found_in_text","path":"utils/infoseek_evaluation_utils.py","file_url":"https://github.com/zhuyjan/WikiSeeker/blob/HEAD/utils/infoseek_evaluation_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"921382b550348607"}},{"code_sha256_prefix":"660ed1d7e213c2a3","entry":"multianswer_exact_match","repo":"zhuyjan/WikiSeeker","repo_kind":"found_in_text","path":"utils/evqa_evaluation_utils.py","file_url":"https://github.com/zhuyjan/WikiSeeker/blob/HEAD/utils/evqa_evaluation_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"660ed1d7e213c2a3"}},{"code_sha256_prefix":"4925192ddc8dfa9a","entry":"normalize_answer","repo":"zhuyjan/WikiSeeker","repo_kind":"found_in_text","path":"utils/infoseek_evaluation_utils.py","file_url":"https://github.com/zhuyjan/WikiSeeker/blob/HEAD/utils/infoseek_evaluation_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4925192ddc8dfa9a"}},{"code_sha256_prefix":"d70aa8c9fa086a8d","entry":"parse_gpu_ids","repo":"zhuyjan/WikiSeeker","repo_kind":"found_in_text","path":"utils/retriever_service.py","file_url":"https://github.com/zhuyjan/WikiSeeker/blob/HEAD/utils/retriever_service.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d70aa8c9fa086a8d"}},{"code_sha256_prefix":"56fd4c74dfbb98ac","entry":"preprocess_answer","repo":"zhuyjan/WikiSeeker","repo_kind":"found_in_text","path":"utils/evqa_evaluation_utils.py","file_url":"https://github.com/zhuyjan/WikiSeeker/blob/HEAD/utils/evqa_evaluation_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"56fd4c74dfbb98ac"}},{"code_sha256_prefix":"2084871add13d2d4","entry":"singleanswer_exact_match","repo":"zhuyjan/WikiSeeker","repo_kind":"found_in_text","path":"utils/evqa_evaluation_utils.py","file_url":"https://github.com/zhuyjan/WikiSeeker/blob/HEAD/utils/evqa_evaluation_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2084871add13d2d4"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}