{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/understanding-the-emergence-of-multimodal","title":"Understanding the Emergence of Multimodal Representation Alignment","arxiv_id":"2502.16282","date":"2025-02-22","proceeding":null,"authors":["Megan Tjandrasuwita","Chanakya Ekbote","Liu Ziyin","Paul Pu Liang"],"abstract":"Multimodal representation learning is fundamentally about transforming incomparable modalities into comparable representations. While prior research primarily focused on explicitly aligning these representations through targeted learning objectives and model architectures, a recent line of work has found that independently trained unimodal models of increasing scale and performance can become implicitly aligned with each other. These findings raise fundamental questions regarding the emergence of aligned representations in multimodal learning. Specifically: (1) when and why does alignment emerge implicitly? and (2) is alignment a reliable indicator of performance? Through a comprehensive empirical investigation, we demonstrate that both the emergence of alignment and its relationship with task performance depend on several critical data characteristics. These include, but are not necessarily limited to, the degree of similarity between the modalities and the balance between redundant and unique information they provide for the task. Our findings suggest that alignment may not be universally beneficial; rather, its impact on performance varies depending on the dataset and task. These insights can help practitioners determine whether increasing alignment between modalities is advantageous or, in some cases, detrimental to achieving optimal performance. Code is released at https://github.com/MeganTj/multimodal_alignment.","url_abs":"https://arxiv.org/abs/2502.16282v1","url_pdf":"https://arxiv.org/pdf/2502.16282v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"understanding-the-emergence-of-multimodal","repo_url":"https://github.com/megantj/multimodal_alignment","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.16282","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.16282"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/megantj/multimodal_alignment","reach":null}],"summary":{"ran_draft_wrong":4,"ran_honours":1},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"254e2c8b9a3bbb86","entry":"get_model_dir","repo":"megantj/multimodal_alignment","repo_kind":"official","path":"scripts/evaluate_multiseed.py","file_url":"https://github.com/megantj/multimodal_alignment/blob/HEAD/scripts/evaluate_multiseed.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"254e2c8b9a3bbb86"}},{"code_sha256_prefix":"2fd383d7796e808a","entry":"get_nrows_ncols","repo":"megantj/multimodal_alignment","repo_kind":"official","path":"scripts/analyze_rus.py","file_url":"https://github.com/megantj/multimodal_alignment/blob/HEAD/scripts/analyze_rus.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2fd383d7796e808a"}},{"code_sha256_prefix":"c8e047d204afc7c6","entry":"get_summary_path","repo":"megantj/multimodal_alignment","repo_kind":"official","path":"scripts/evaluate_multiseed.py","file_url":"https://github.com/megantj/multimodal_alignment/blob/HEAD/scripts/evaluate_multiseed.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c8e047d204afc7c6"}},{"code_sha256_prefix":"1fc9c17cf57ea536","entry":"parse_str","repo":"megantj/multimodal_alignment","repo_kind":"official","path":"scripts/analyze_rus.py","file_url":"https://github.com/megantj/multimodal_alignment/blob/HEAD/scripts/analyze_rus.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1fc9c17cf57ea536"}},{"code_sha256_prefix":"ce63bcb39d0499be","entry":"plot_scatter","repo":"megantj/multimodal_alignment","repo_kind":"official","path":"scripts/analyze_rus.py","file_url":"https://github.com/megantj/multimodal_alignment/blob/HEAD/scripts/analyze_rus.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ce63bcb39d0499be"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}