{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2510-14605","title":"Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering","arxiv_id":"2510.14605","date":"2025-10-16","proceeding":"NeurIPS","authors":["Yuyang Hong","Jiaqi Gu","Qi Yang","Lubin Fan","Yue Wu","Ying Wang","Kun Ding","Shiming Xiang","Jieping Ye"],"abstract":"Knowledge-based visual question answering (KB-VQA) requires visual language models (VLMs) to integrate visual understanding with external knowledge retrieval. Although retrieval-augmented generation (RAG) achieves significant advances in this task by combining knowledge-base querying, it still struggles with the quality of multimodal queries and the relevance of retrieved results. To overcome these challenges, we propose a novel three-stage method, termed Wiki-PRF, including Processing, Retrieval and Filtering stages. The processing stage dynamically invokes visual tools to extract precise multimodal information for retrieval. The retrieval stage integrates visual and text features to achieve multimodal knowledge retrieval. The filtering stage performs relevance filtering and concentration on retrieval results. To this end, we introduce a visual language model trained with answer accuracy and format consistency as reward signals via a reinforcement learning manner. This enhances the model's reasoning, tool invocation for accurate queries, and filtering of irrelevant content. Experiments on benchmark datasets (E-VQA and InfoSeek) show significant improvements~(36.0 and 42.8) in answer quality, achieving state-of-the-art performance. Code is available at https://github.com/cqu-student/Wiki-PRF","url_abs":"https://arxiv.org/abs/2510.14605","url_pdf":"https://arxiv.org/pdf/2510.14605","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2510.14605","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2510.14605"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/om-ai-lab/VLM-R1","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"ran_fixture":1,"unverified":7},"by_repo_kind":{"found_in_text":{"samples":9,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"57e95166ade5f25c","entry":"build_transform","repo":"om-ai-lab/VLM-R1","repo_kind":"found_in_text","path":"src/open-r1-multimodal/src/open_r1/vlm_modules/internvl_module.py","file_url":"https://github.com/om-ai-lab/VLM-R1/blob/HEAD/src/open-r1-multimodal/src/open_r1/vlm_modules/internvl_module.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"57e95166ade5f25c"}},{"code_sha256_prefix":"3f31b48fd862935d","entry":"get_image_data_url","repo":"om-ai-lab/VLM-R1","repo_kind":"found_in_text","path":"src/open-r1-multimodal/local_scripts/create_vision_cot_data.py","file_url":"https://github.com/om-ai-lab/VLM-R1/blob/HEAD/src/open-r1-multimodal/local_scripts/create_vision_cot_data.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3f31b48fd862935d"}},{"code_sha256_prefix":"41c548a25e90eb39","entry":"extract_problem_solution","repo":"om-ai-lab/VLM-R1","repo_kind":"found_in_text","path":"src/open-r1-multimodal/local_scripts/prepare_hf_data.py","file_url":"https://github.com/om-ai-lab/VLM-R1/blob/HEAD/src/open-r1-multimodal/local_scripts/prepare_hf_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"41c548a25e90eb39"}},{"code_sha256_prefix":"dd232d5d1e2dcf5c","entry":"extract_system_message","repo":"om-ai-lab/VLM-R1","repo_kind":"found_in_text","path":"src/open-r1-multimodal/src/open_r1/vlm_modules/internvl_module.py","file_url":"https://github.com/om-ai-lab/VLM-R1/blob/HEAD/src/open-r1-multimodal/src/open_r1/vlm_modules/internvl_module.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dd232d5d1e2dcf5c"}},{"code_sha256_prefix":"e89f2eec0a18d4c2","entry":"format_reward","repo":"om-ai-lab/VLM-R1","repo_kind":"found_in_text","path":"src/open-r1-multimodal/src/open_r1/grpo.py","file_url":"https://github.com/om-ai-lab/VLM-R1/blob/HEAD/src/open-r1-multimodal/src/open_r1/grpo.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e89f2eec0a18d4c2"}},{"code_sha256_prefix":"58ab41ce28a94c62","entry":"gpt4o_query","repo":"om-ai-lab/VLM-R1","repo_kind":"found_in_text","path":"src/open-r1-multimodal/local_scripts/create_vision_cot_data.py","file_url":"https://github.com/om-ai-lab/VLM-R1/blob/HEAD/src/open-r1-multimodal/local_scripts/create_vision_cot_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"58ab41ce28a94c62"}},{"code_sha256_prefix":"bdcc94560fb87f30","entry":"load_image_from_path","repo":"om-ai-lab/VLM-R1","repo_kind":"found_in_text","path":"src/open-r1-multimodal/local_scripts/prepare_hf_data.py","file_url":"https://github.com/om-ai-lab/VLM-R1/blob/HEAD/src/open-r1-multimodal/local_scripts/prepare_hf_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bdcc94560fb87f30"}},{"code_sha256_prefix":"caa7fed63b8d486c","entry":"process_conversation_list","repo":"om-ai-lab/VLM-R1","repo_kind":"found_in_text","path":"src/open-r1-multimodal/src/open_r1/vlm_modules/internvl_module.py","file_url":"https://github.com/om-ai-lab/VLM-R1/blob/HEAD/src/open-r1-multimodal/src/open_r1/vlm_modules/internvl_module.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"caa7fed63b8d486c"}},{"code_sha256_prefix":"f029d6b5f19d2885","entry":"process_single_item","repo":"om-ai-lab/VLM-R1","repo_kind":"found_in_text","path":"src/open-r1-multimodal/local_scripts/create_vision_cot_data.py","file_url":"https://github.com/om-ai-lab/VLM-R1/blob/HEAD/src/open-r1-multimodal/local_scripts/create_vision_cot_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f029d6b5f19d2885"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CV","source":"arxiv_api"},"syntology_extracted_results":null}