{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/provision-programmatically-scaling-vision","title":"ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Models","arxiv_id":"2412.07012","date":"2024-12-09","proceeding":null,"authors":["Jieyu Zhang","Le Xue","Linxin Song","Jun Wang","Weikai Huang","Manli Shu","An Yan","Zixian Ma","Juan Carlos Niebles","Silvio Savarese","Caiming Xiong","Zeyuan Chen","Ranjay Krishna","ran Xu"],"abstract":"With the rise of multimodal applications, instruction data has become critical for training multimodal language models capable of understanding complex image-based queries. Existing practices rely on powerful but costly large language models (LLMs) or multimodal language models (MLMs) to produce instruction data. These are often prone to hallucinations, licensing issues and the generation process is often hard to scale and interpret. In this work, we present a programmatic approach that employs scene graphs as symbolic representations of images and human-written programs to systematically synthesize vision-centric instruction data. Our approach ensures the interpretability and controllability of the data generation process and scales efficiently while maintaining factual accuracy. By implementing a suite of 24 single-image, 14 multi-image instruction generators, and a scene graph generation pipeline, we build a scalable, cost-effective system: ProVision which produces diverse question-answer pairs concerning objects, attributes, relations, depth, etc., for any given image. Applied to Visual Genome and DataComp datasets, we generate over 10 million instruction data points, ProVision-10M, and leverage them in both pretraining and instruction tuning stages of MLMs. When adopted in the instruction tuning stage, our single-image instruction data yields up to a 7% improvement on the 2D split and 8% on the 3D split of CVBench, along with a 3% increase in performance on QBench2, RealWorldQA, and MMMU. Our multi-image instruction data leads to an 8% improvement on Mantis-Eval. Incorporation of our data in both pre-training and fine-tuning stages of xGen-MM-4B leads to an averaged improvement of 1.6% across 11 benchmarks.","url_abs":"https://arxiv.org/abs/2412.07012v3","url_pdf":"https://arxiv.org/pdf/2412.07012v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"provision-programmatically-scaling-vision","repo_url":"https://github.com/jieyuz2/provision","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"graph-generation","task_name":"Graph Generation"},{"task_slug":"scene-graph-generation","task_name":"Scene Graph Generation"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-question-answering-on-mm-vet","task":"Visual Question Answering","dataset":"MM-Vet","model":"LLaVA-1.5-7B (VG-S)","rank_in_archive_order":107,"of":231,"metrics":{"GPT-4 score":"40.4"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-mm-vet","task":"Visual Question Answering","dataset":"MM-Vet","model":"LLaVA-1.5-7B (DC-S)","rank_in_archive_order":123,"of":231,"metrics":{"GPT-4 score":"38.5"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.07012","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.07012"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jieyuz2/provision","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":15},"by_repo_kind":{"official":{"samples":15,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1b9b3ba31d9f08f9","entry":"boxOverlap","repo":"jieyuz2/provision","repo_kind":"official","path":"provision/dataset.py","file_url":"https://github.com/jieyuz2/provision/blob/HEAD/provision/dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1b9b3ba31d9f08f9"}},{"code_sha256_prefix":"7f28431ff5823dfe","entry":"cate_grounding","repo":"jieyuz2/provision","repo_kind":"official","path":"provision/annotation/relation_grounding.py","file_url":"https://github.com/jieyuz2/provision/blob/HEAD/provision/annotation/relation_grounding.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7f28431ff5823dfe"}},{"code_sha256_prefix":"7a269cd987117ade","entry":"depth_estimation","repo":"jieyuz2/provision","repo_kind":"official","path":"provision/annotation/depth_estimation.py","file_url":"https://github.com/jieyuz2/provision/blob/HEAD/provision/annotation/depth_estimation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7a269cd987117ade"}},{"code_sha256_prefix":"3010b213fcb650e6","entry":"get_attributes","repo":"jieyuz2/provision","repo_kind":"official","path":"provision/annotation/attribute_llava.py","file_url":"https://github.com/jieyuz2/provision/blob/HEAD/provision/annotation/attribute_llava.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3010b213fcb650e6"}},{"code_sha256_prefix":"4354809003526c37","entry":"get_image","repo":"jieyuz2/provision","repo_kind":"official","path":"provision/annotation/utils.py","file_url":"https://github.com/jieyuz2/provision/blob/HEAD/provision/annotation/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4354809003526c37"}},{"code_sha256_prefix":"5f90221712cb63f4","entry":"load_image","repo":"jieyuz2/provision","repo_kind":"official","path":"provision/annotation/attribute_llava.py","file_url":"https://github.com/jieyuz2/provision/blob/HEAD/provision/annotation/attribute_llava.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5f90221712cb63f4"}},{"code_sha256_prefix":"a72c327b29ab95e1","entry":"normalize_attributes","repo":"jieyuz2/provision","repo_kind":"official","path":"provision/dataset.py","file_url":"https://github.com/jieyuz2/provision/blob/HEAD/provision/dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a72c327b29ab95e1"}},{"code_sha256_prefix":"3cfade61c1d29398","entry":"normalize_relations","repo":"jieyuz2/provision","repo_kind":"official","path":"provision/dataset.py","file_url":"https://github.com/jieyuz2/provision/blob/HEAD/provision/dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3cfade61c1d29398"}},{"code_sha256_prefix":"a13bc543209b74ab","entry":"object_detection","repo":"jieyuz2/provision","repo_kind":"official","path":"provision/annotation/object_detection.py","file_url":"https://github.com/jieyuz2/provision/blob/HEAD/provision/annotation/object_detection.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a13bc543209b74ab"}},{"code_sha256_prefix":"a1e26f63b595bca0","entry":"rel_parsing","repo":"jieyuz2/provision","repo_kind":"official","path":"provision/annotation/relation_grounding.py","file_url":"https://github.com/jieyuz2/provision/blob/HEAD/provision/annotation/relation_grounding.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a1e26f63b595bca0"}},{"code_sha256_prefix":"f69b840efc02bd76","entry":"sam_mask_generation","repo":"jieyuz2/provision","repo_kind":"official","path":"provision/annotation/segmentation.py","file_url":"https://github.com/jieyuz2/provision/blob/HEAD/provision/annotation/segmentation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f69b840efc02bd76"}},{"code_sha256_prefix":"8b3900eb71dae6ba","entry":"sam_mask_prediction","repo":"jieyuz2/provision","repo_kind":"official","path":"provision/annotation/segmentation.py","file_url":"https://github.com/jieyuz2/provision/blob/HEAD/provision/annotation/segmentation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8b3900eb71dae6ba"}},{"code_sha256_prefix":"9008ca38b586fa68","entry":"save_depth_res","repo":"jieyuz2/provision","repo_kind":"official","path":"provision/annotation/depth_estimation.py","file_url":"https://github.com/jieyuz2/provision/blob/HEAD/provision/annotation/depth_estimation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9008ca38b586fa68"}},{"code_sha256_prefix":"b67cc1d3454644b8","entry":"save_mask_as_png","repo":"jieyuz2/provision","repo_kind":"official","path":"provision/annotation/utils.py","file_url":"https://github.com/jieyuz2/provision/blob/HEAD/provision/annotation/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b67cc1d3454644b8"}},{"code_sha256_prefix":"2ac58005d7f320e4","entry":"save_sep_mask","repo":"jieyuz2/provision","repo_kind":"official","path":"provision/annotation/utils.py","file_url":"https://github.com/jieyuz2/provision/blob/HEAD/provision/annotation/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2ac58005d7f320e4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}