{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-12960","title":"DiM\\textsuperscript{3}: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Merging","arxiv_id":"2605.12960","date":"2026-05-13","proceeding":null,"authors":["Zijing Wang","Mingyang Wang","Ercong Nie","Yongkang Liu","Shi Feng","Mengjie Zhao","Daling Wang","Xiaocui Yang","Hinrich Schütze"],"abstract":"Towards more general and human-like intelligence, large language models should seamlessly integrate both multilingual and multimodal capabilities; however, extending an existing multimodal model to many languages typically requires expensive multilingual multimodal data construction and repeated end-to-end retraining. We study a training-free alternative: injecting multilingual capability into an existing multimodal model by composing residual updates in the shared language model backbone. The key challenge is that multilingual and multimodal updates are heterogeneous, reflecting different functional roles in the shared model. To address this, we propose Direction- and Magnitude-aware Multilingual Multimodal merging (DiM3), which selectively composes the two updates at each parameter dimension while preserving the original vision encoder and multimodal projector. Experiments on multilingual benchmarks in both text-only and vision-language settings, covering 57 languages across LLaVA- and Qwen-based backbones, show that DiM3 consistently outperforms existing merging baselines, substantially improves multilingual performance over the original multimodal model, and remains competitive with dedicated multilingual multimodal fine-tuning while largely retaining general multimodal ability. We further show that DiM3 can be directly applied to already trained multilingual multimodal models and still yield additional gains. Further interpretability analysis shows that DiM3 primarily reshapes intermediate-layer semantic representations, strengthening cross-lingual alignment under both text-only and multimodal inputs while preserving higher-layer task-sensitive structure. Our repository is on https://github.com/wzj1718/DiM3.","url_abs":"https://arxiv.org/abs/2605.12960","url_pdf":"https://arxiv.org/pdf/2605.12960","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2605.12960","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.12960"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/wzj1718/DiM3","reach":null}],"summary":{"ran":8,"ran_honours":1},"by_repo_kind":{"found_in_text":{"samples":9,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"7e60730f07a2532a","entry":"act","repo":"wzj1718/DiM3","repo_kind":"found_in_text","path":"merge/merging_methods/pcb_merging.py","file_url":"https://github.com/wzj1718/DiM3/blob/HEAD/merge/merging_methods/pcb_merging.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7e60730f07a2532a"}},{"code_sha256_prefix":"860bf587f2fbf4e0","entry":"add_ptm_to_tv","repo":"wzj1718/DiM3","repo_kind":"found_in_text","path":"merge/merging_methods/ties_merging_utils.py","file_url":"https://github.com/wzj1718/DiM3/blob/HEAD/merge/merging_methods/ties_merging_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"860bf587f2fbf4e0"}},{"code_sha256_prefix":"5a7ea2d1b13f410f","entry":"check_state_dicts_equal","repo":"wzj1718/DiM3","repo_kind":"found_in_text","path":"merge/merging_methods/ties_merging_utils.py","file_url":"https://github.com/wzj1718/DiM3/blob/HEAD/merge/merging_methods/ties_merging_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5a7ea2d1b13f410f"}},{"code_sha256_prefix":"76418510bf7df06d","entry":"clamp","repo":"wzj1718/DiM3","repo_kind":"found_in_text","path":"merge/merging_methods/pcb_merging.py","file_url":"https://github.com/wzj1718/DiM3/blob/HEAD/merge/merging_methods/pcb_merging.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"76418510bf7df06d"}},{"code_sha256_prefix":"c8856c78385f817a","entry":"flatten_ckpt_into_vec","repo":"wzj1718/DiM3","repo_kind":"found_in_text","path":"merge/merging_methods/utils.py","file_url":"https://github.com/wzj1718/DiM3/blob/HEAD/merge/merging_methods/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c8856c78385f817a"}},{"code_sha256_prefix":"fd399befaee47739","entry":"get_task_vector","repo":"wzj1718/DiM3","repo_kind":"found_in_text","path":"merge/merging_methods/utils.py","file_url":"https://github.com/wzj1718/DiM3/blob/HEAD/merge/merging_methods/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fd399befaee47739"}},{"code_sha256_prefix":"657972dcf33a3093","entry":"normalize","repo":"wzj1718/DiM3","repo_kind":"found_in_text","path":"merge/merging_methods/pcb_merging.py","file_url":"https://github.com/wzj1718/DiM3/blob/HEAD/merge/merging_methods/pcb_merging.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"657972dcf33a3093"}},{"code_sha256_prefix":"39ddec4bec43fc72","entry":"select_trainable_params","repo":"wzj1718/DiM3","repo_kind":"found_in_text","path":"merge/merging_methods/utils.py","file_url":"https://github.com/wzj1718/DiM3/blob/HEAD/merge/merging_methods/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39ddec4bec43fc72"}},{"code_sha256_prefix":"72391ca34e9a8cfe","entry":"state_dict_to_vector","repo":"wzj1718/DiM3","repo_kind":"found_in_text","path":"merge/merging_methods/ties_merging_utils.py","file_url":"https://github.com/wzj1718/DiM3/blob/HEAD/merge/merging_methods/ties_merging_utils.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"72391ca34e9a8cfe"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}