{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/visualwebinstruct-scaling-up-multimodal","title":"VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Search","arxiv_id":"2503.10582","date":"2025-03-13","proceeding":null,"authors":["Yiming Jia","Jiachen Li","Xiang Yue","Bo Li","Ping Nie","Kai Zou","Wenhu Chen"],"abstract":"Vision-Language Models have made significant progress on many perception-focused tasks, however, their progress on reasoning-focused tasks seem to be limited due to the lack of high-quality and diverse training data. In this work, we aim to address the scarcity issue of reasoning-focused multimodal datasets. We propose VisualWebInstruct - a novel approach that leverages search engine to create a diverse, and high-quality dataset spanning multiple disciplines like math, physics, finance, chemistry, etc. Starting with meticulously selected 30,000 seed images, we employ Google Image search to identify websites containing similar images. We collect and process the HTMLs from over 700K unique URL sources. Through a pipeline of content extraction, filtering and synthesis, we build a dataset of approximately 900K question-answer pairs, with 40% being visual QA pairs and the rest as text QA pairs. Models fine-tuned on VisualWebInstruct demonstrate significant performance gains: (1) training from Llava-OV-mid shows 10-20% absolute point gains across benchmarks, (2) training from MAmmoTH-VL shows 5% absoluate gain. Our best model MAmmoTH-VL2 shows state-of-the-art performance within the 10B parameter class on MMMU-Pro-std (40.7%), MathVerse (42.6%), and DynaMath (55.7%). These remarkable results highlight the effectiveness of our dataset in enhancing VLMs' reasoning capabilities for complex multimodal tasks.","url_abs":"https://arxiv.org/abs/2503.10582v1","url_pdf":"https://arxiv.org/pdf/2503.10582v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"visualwebinstruct-scaling-up-multimodal","repo_url":"https://github.com/tiger-ai-lab/visualwebinstruct","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"math","task_name":"Math"}],"methods":[{"method_slug":"url","method_name":"URL"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.10582","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.10582"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tiger-ai-lab/visualwebinstruct","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":13},"by_repo_kind":{"listed":{"samples":13,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0a9f64be9528b7e8","entry":"create_index_map","repo":"tiger-ai-lab/visualwebinstruct","repo_kind":"listed","path":"VisualWebInstruct/answer_alignment.py","file_url":"https://github.com/tiger-ai-lab/visualwebinstruct/blob/HEAD/VisualWebInstruct/answer_alignment.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a9f64be9528b7e8"}},{"code_sha256_prefix":"56076c2e87259345","entry":"detect_base_url","repo":"tiger-ai-lab/visualwebinstruct","repo_kind":"listed","path":"VisualWebInstruct/acc_tree.py","file_url":"https://github.com/tiger-ai-lab/visualwebinstruct/blob/HEAD/VisualWebInstruct/acc_tree.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"56076c2e87259345"}},{"code_sha256_prefix":"2d25b921e96b5b59","entry":"extract_content","repo":"tiger-ai-lab/visualwebinstruct","repo_kind":"listed","path":"VisualWebInstruct/acc_tree.py","file_url":"https://github.com/tiger-ai-lab/visualwebinstruct/blob/HEAD/VisualWebInstruct/acc_tree.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2d25b921e96b5b59"}},{"code_sha256_prefix":"b445363cea20a105","entry":"extract_content_from_response","repo":"tiger-ai-lab/visualwebinstruct","repo_kind":"listed","path":"VisualWebInstruct/qa_extraction/qa.py","file_url":"https://github.com/tiger-ai-lab/visualwebinstruct/blob/HEAD/VisualWebInstruct/qa_extraction/qa.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b445363cea20a105"}},{"code_sha256_prefix":"ff37e5c1bf683283","entry":"format_qa_output","repo":"tiger-ai-lab/visualwebinstruct","repo_kind":"listed","path":"VisualWebInstruct/qa_extraction/qa.py","file_url":"https://github.com/tiger-ai-lab/visualwebinstruct/blob/HEAD/VisualWebInstruct/qa_extraction/qa.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ff37e5c1bf683283"}},{"code_sha256_prefix":"e5faa0db66429c23","entry":"grouper","repo":"tiger-ai-lab/visualwebinstruct","repo_kind":"listed","path":"VisualWebInstruct/html_download.py","file_url":"https://github.com/tiger-ai-lab/visualwebinstruct/blob/HEAD/VisualWebInstruct/html_download.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e5faa0db66429c23"}},{"code_sha256_prefix":"9db5b97b78099e67","entry":"load_huggingface_dataset","repo":"tiger-ai-lab/visualwebinstruct","repo_kind":"listed","path":"VisualWebInstruct/answer_alignment.py","file_url":"https://github.com/tiger-ai-lab/visualwebinstruct/blob/HEAD/VisualWebInstruct/answer_alignment.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9db5b97b78099e67"}},{"code_sha256_prefix":"cd1b2e17d5682457","entry":"load_jsonl","repo":"tiger-ai-lab/visualwebinstruct","repo_kind":"listed","path":"VisualWebInstruct/answer_alignment.py","file_url":"https://github.com/tiger-ai-lab/visualwebinstruct/blob/HEAD/VisualWebInstruct/answer_alignment.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cd1b2e17d5682457"}},{"code_sha256_prefix":"27e7c637bc8bf1a4","entry":"normalize_url","repo":"tiger-ai-lab/visualwebinstruct","repo_kind":"listed","path":"VisualWebInstruct/deduplication.py","file_url":"https://github.com/tiger-ai-lab/visualwebinstruct/blob/HEAD/VisualWebInstruct/deduplication.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"27e7c637bc8bf1a4"}},{"code_sha256_prefix":"8849845b24577ba4","entry":"parse_markdown_qa","repo":"tiger-ai-lab/visualwebinstruct","repo_kind":"listed","path":"VisualWebInstruct/qa_extraction/qa.py","file_url":"https://github.com/tiger-ai-lab/visualwebinstruct/blob/HEAD/VisualWebInstruct/qa_extraction/qa.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8849845b24577ba4"}},{"code_sha256_prefix":"caa6ec9df9fd4678","entry":"process_batch","repo":"tiger-ai-lab/visualwebinstruct","repo_kind":"listed","path":"VisualWebInstruct/deduplication.py","file_url":"https://github.com/tiger-ai-lab/visualwebinstruct/blob/HEAD/VisualWebInstruct/deduplication.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"caa6ec9df9fd4678"}},{"code_sha256_prefix":"bbedc76fe0bce10b","entry":"process_single_file","repo":"tiger-ai-lab/visualwebinstruct","repo_kind":"listed","path":"VisualWebInstruct/deduplication.py","file_url":"https://github.com/tiger-ai-lab/visualwebinstruct/blob/HEAD/VisualWebInstruct/deduplication.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bbedc76fe0bce10b"}},{"code_sha256_prefix":"f535e5b470ab1c6b","entry":"should_keep_element","repo":"tiger-ai-lab/visualwebinstruct","repo_kind":"listed","path":"VisualWebInstruct/acc_tree.py","file_url":"https://github.com/tiger-ai-lab/visualwebinstruct/blob/HEAD/VisualWebInstruct/acc_tree.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f535e5b470ab1c6b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}