{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reconstructive-visual-instruction-tuning","title":"Reconstructive Visual Instruction Tuning","arxiv_id":"2410.09575","date":"2024-10-12","proceeding":null,"authors":["Haochen Wang","Anlin Zheng","Yucheng Zhao","Tiancai Wang","Zheng Ge","Xiangyu Zhang","Zhaoxiang Zhang"],"abstract":"This paper introduces reconstructive visual instruction tuning (ROSS), a family of Large Multimodal Models (LMMs) that exploit vision-centric supervision signals. In contrast to conventional visual instruction tuning approaches that exclusively supervise text outputs, ROSS prompts LMMs to supervise visual outputs via reconstructing input images. By doing so, it capitalizes on the inherent richness and detail present within input images themselves, which are often lost in pure text supervision. However, producing meaningful feedback from natural images is challenging due to the heavy spatial redundancy of visual signals. To address this issue, ROSS employs a denoising objective to reconstruct latent representations of input images, avoiding directly regressing exact raw RGB values. This intrinsic activation design inherently encourages LMMs to maintain image detail, thereby enhancing their fine-grained comprehension capabilities and reducing hallucinations. Empirically, ROSS consistently brings significant improvements across different visual encoders and language models. In comparison with extrinsic assistance state-of-the-art alternatives that aggregate multiple visual experts, ROSS delivers competitive performance with a single SigLIP visual encoder, demonstrating the efficacy of our vision-centric supervision tailored for visual outputs.","url_abs":"https://arxiv.org/abs/2410.09575v2","url_pdf":"https://arxiv.org/pdf/2410.09575v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reconstructive-visual-instruction-tuning","repo_url":"https://github.com/haochen-wang409/ross","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.09575","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.09575"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/haochen-wang409/ross","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3,"ran_fixture":4,"unverified":6},"by_repo_kind":{"listed":{"samples":13,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7e03b180fa317c9a","entry":"divide_to_patches","repo":"haochen-wang409/ross","repo_kind":"listed","path":"ross/mm_utils.py","file_url":"https://github.com/haochen-wang409/ross/blob/HEAD/ross/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7e03b180fa317c9a"}},{"code_sha256_prefix":"5c8ea92d43ead77d","entry":"get_1d_sincos_pos_embed_from_grid","repo":"haochen-wang409/ross","repo_kind":"listed","path":"ross/model/utils.py","file_url":"https://github.com/haochen-wang409/ross/blob/HEAD/ross/model/utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5c8ea92d43ead77d"}},{"code_sha256_prefix":"77e8a3ac46f3afec","entry":"get_2d_sincos_pos_embed","repo":"haochen-wang409/ross","repo_kind":"listed","path":"ross/model/utils.py","file_url":"https://github.com/haochen-wang409/ross/blob/HEAD/ross/model/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"77e8a3ac46f3afec"}},{"code_sha256_prefix":"468eedeba67f1b00","entry":"resize_and_pad_image","repo":"haochen-wang409/ross","repo_kind":"listed","path":"ross/mm_utils.py","file_url":"https://github.com/haochen-wang409/ross/blob/HEAD/ross/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"468eedeba67f1b00"}},{"code_sha256_prefix":"3999ff487573f32c","entry":"select_best_resolution","repo":"haochen-wang409/ross","repo_kind":"listed","path":"ross/mm_utils.py","file_url":"https://github.com/haochen-wang409/ross/blob/HEAD/ross/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3999ff487573f32c"}},{"code_sha256_prefix":"10893c4608c08075","entry":"split_to_even_chunks","repo":"haochen-wang409/ross","repo_kind":"listed","path":"ross/ross_trainer.py","file_url":"https://github.com/haochen-wang409/ross/blob/HEAD/ross/ross_trainer.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"10893c4608c08075"}},{"code_sha256_prefix":"55c32993da87759b","entry":"unpad_image","repo":"haochen-wang409/ross","repo_kind":"listed","path":"ross/model/ross_arch.py","file_url":"https://github.com/haochen-wang409/ross/blob/HEAD/ross/model/ross_arch.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"55c32993da87759b"}},{"code_sha256_prefix":"25d7c8f510336006","entry":"count_consecutive_blocks","repo":"haochen-wang409/ross","repo_kind":"listed","path":"ross/model/ross_arch.py","file_url":"https://github.com/haochen-wang409/ross/blob/HEAD/ross/model/ross_arch.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"25d7c8f510336006"}},{"code_sha256_prefix":"2884deadbb637c3e","entry":"get_2d_sincos_pos_embed_from_grid","repo":"haochen-wang409/ross","repo_kind":"listed","path":"ross/model/utils.py","file_url":"https://github.com/haochen-wang409/ross/blob/HEAD/ross/model/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2884deadbb637c3e"}},{"code_sha256_prefix":"bb35e3ac741bb2c9","entry":"get_mm_adapter_state_maybe_zero_3","repo":"haochen-wang409/ross","repo_kind":"listed","path":"ross/ross_trainer.py","file_url":"https://github.com/haochen-wang409/ross/blob/HEAD/ross/ross_trainer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bb35e3ac741bb2c9"}},{"code_sha256_prefix":"735025744c1ab0cf","entry":"maybe_zero_3","repo":"haochen-wang409/ross","repo_kind":"listed","path":"ross/ross_trainer.py","file_url":"https://github.com/haochen-wang409/ross/blob/HEAD/ross/ross_trainer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"735025744c1ab0cf"}},{"code_sha256_prefix":"37899f22fb191b37","entry":"pretty_print_semaphore","repo":"haochen-wang409/ross","repo_kind":"listed","path":"ross/utils.py","file_url":"https://github.com/haochen-wang409/ross/blob/HEAD/ross/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"37899f22fb191b37"}},{"code_sha256_prefix":"f9939a84b9a65279","entry":"violates_moderation","repo":"haochen-wang409/ross","repo_kind":"listed","path":"ross/utils.py","file_url":"https://github.com/haochen-wang409/ross/blob/HEAD/ross/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f9939a84b9a65279"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}