{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bring-reason-to-vision-understanding","title":"Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging","arxiv_id":"2505.05464","date":"2025-05-08","proceeding":null,"authors":["Shiqi Chen","Jinghan Zhang","Tongyao Zhu","Wei Liu","Siyang Gao","Miao Xiong","Manling Li","Junxian He"],"abstract":"Vision-Language Models (VLMs) combine visual perception with the general capabilities, such as reasoning, of Large Language Models (LLMs). However, the mechanisms by which these two abilities can be combined and contribute remain poorly understood. In this work, we explore to compose perception and reasoning through model merging that connects parameters of different models. Unlike previous works that often focus on merging models of the same kind, we propose merging models across modalities, enabling the incorporation of the reasoning capabilities of LLMs into VLMs. Through extensive experiments, we demonstrate that model merging offers a successful pathway to transfer reasoning abilities from LLMs to VLMs in a training-free manner. Moreover, we utilize the merged models to understand the internal mechanism of perception and reasoning and how merging affects it. We find that perception capabilities are predominantly encoded in the early layers of the model, whereas reasoning is largely facilitated by the middle-to-late layers. After merging, we observe that all layers begin to contribute to reasoning, whereas the distribution of perception abilities across layers remains largely unchanged. These observations shed light on the potential of model merging as a tool for multimodal integration and interpretation.","url_abs":"https://arxiv.org/abs/2505.05464v1","url_pdf":"https://arxiv.org/pdf/2505.05464v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bring-reason-to-vision-understanding","repo_url":"https://github.com/shiqichen17/vlm_merging","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2505.05464","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.05464"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/shiqichen17/VLM","reach":{"status":"gone","observed_at":"2026-09-16","how":"tree_404+repo_404"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shiqichen17/vlm_merging","reach":null}],"summary":{"ran_draft_wrong":5,"ran_fixture":4,"ran_honours":1,"unverified":1},"by_repo_kind":{"official":{"samples":11,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"b46f6a6e9e499e42","entry":"calculate_majority_sign_mask","repo":"shiqichen17/vlm_merging","repo_kind":"official","path":"merge.py","file_url":"https://github.com/shiqichen17/vlm_merging/blob/HEAD/merge.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b46f6a6e9e499e42"}},{"code_sha256_prefix":"a2577df9ddbdf691","entry":"dare_linear","repo":"shiqichen17/vlm_merging","repo_kind":"official","path":"merge.py","file_url":"https://github.com/shiqichen17/vlm_merging/blob/HEAD/merge.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a2577df9ddbdf691"}},{"code_sha256_prefix":"90236229f79c5ceb","entry":"dare_ties","repo":"shiqichen17/vlm_merging","repo_kind":"official","path":"merge.py","file_url":"https://github.com/shiqichen17/vlm_merging/blob/HEAD/merge.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"90236229f79c5ceb"}},{"code_sha256_prefix":"9eeb21ee73863b3b","entry":"disjoint_merge","repo":"shiqichen17/vlm_merging","repo_kind":"official","path":"merge.py","file_url":"https://github.com/shiqichen17/vlm_merging/blob/HEAD/merge.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9eeb21ee73863b3b"}},{"code_sha256_prefix":"01e735fa95b0a1a9","entry":"extract_layer_number","repo":"shiqichen17/vlm_merging","repo_kind":"official","path":"merge.py","file_url":"https://github.com/shiqichen17/vlm_merging/blob/HEAD/merge.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"01e735fa95b0a1a9"}},{"code_sha256_prefix":"a527bf2700a78e15","entry":"magnitude_based_pruning","repo":"shiqichen17/vlm_merging","repo_kind":"official","path":"merge.py","file_url":"https://github.com/shiqichen17/vlm_merging/blob/HEAD/merge.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a527bf2700a78e15"}},{"code_sha256_prefix":"6caa545d200ee863","entry":"prune","repo":"shiqichen17/vlm_merging","repo_kind":"official","path":"merge.py","file_url":"https://github.com/shiqichen17/vlm_merging/blob/HEAD/merge.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6caa545d200ee863"}},{"code_sha256_prefix":"37b2d5955b59dbdd","entry":"random_pruning","repo":"shiqichen17/vlm_merging","repo_kind":"official","path":"merge.py","file_url":"https://github.com/shiqichen17/vlm_merging/blob/HEAD/merge.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"37b2d5955b59dbdd"}},{"code_sha256_prefix":"96438568fea309ce","entry":"reshape_weight_task_tensors","repo":"shiqichen17/vlm_merging","repo_kind":"official","path":"merge.py","file_url":"https://github.com/shiqichen17/vlm_merging/blob/HEAD/merge.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"96438568fea309ce"}},{"code_sha256_prefix":"0683b00fee94ac33","entry":"ties","repo":"shiqichen17/vlm_merging","repo_kind":"official","path":"merge.py","file_url":"https://github.com/shiqichen17/vlm_merging/blob/HEAD/merge.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0683b00fee94ac33"}},{"code_sha256_prefix":"f2ca9ea81cadf810","entry":"merge_models","repo":"shiqichen17/vlm_merging","repo_kind":"official","path":"merge.py","file_url":"https://github.com/shiqichen17/vlm_merging/blob/HEAD/merge.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f2ca9ea81cadf810"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}