{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/seeing-is-understanding-unlocking-causal","title":"Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs","arxiv_id":"2503.02597","date":"2025-03-04","proceeding":null,"authors":["Wei-Yao Wang","Zhao Wang","Helen Suzuki","Yoshiyuki Kobayashi"],"abstract":"Recent Multimodal Large Language Models (MLLMs) have demonstrated significant progress in perceiving and reasoning over multimodal inquiries, ushering in a new research era for foundation models. However, vision-language misalignment in MLLMs has emerged as a critical challenge, where the textual responses generated by these models are not factually aligned with the given text-image inputs. Existing efforts to address vision-language misalignment have focused on developing specialized vision-language connectors or leveraging visual instruction tuning from diverse domains. In this paper, we tackle this issue from a fundamental yet unexplored perspective by revisiting the core architecture of MLLMs. Most MLLMs are typically built on decoder-only LLMs consisting of a causal attention mechanism, which limits the ability of earlier modalities (e.g., images) to incorporate information from later modalities (e.g., text). To address this problem, we propose AKI, a novel MLLM that unlocks causal attention into modality-mutual attention (MMA) to enable image tokens to attend to text tokens. This simple yet effective design allows AKI to achieve superior performance in 12 multimodal understanding benchmarks (+7.2% on average) without introducing additional parameters and increasing training time. Our MMA design is intended to be generic, allowing for application across various modalities, and scalable to accommodate diverse multimodal scenarios. The code is publicly available at https://github.com/sony/aki, and we will release our AKI-4B model to encourage further advancements in MLLMs across various directions.","url_abs":"https://arxiv.org/abs/2503.02597v1","url_pdf":"https://arxiv.org/pdf/2503.02597v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"seeing-is-understanding-unlocking-causal","repo_url":"https://github.com/sony/aki","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.02597","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.02597"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/sony/aki","reach":null}],"summary":{"ran":3,"ran_draft_wrong":1,"ran_honours":2,"unverified":6},"by_repo_kind":{"official":{"samples":12,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"fba7faec475b6155","entry":"DecoupledEmbedding","repo":"sony/aki","repo_kind":"official","path":"codes/open_flamingo/src/modeling_aki.py","file_url":"https://github.com/sony/aki/blob/HEAD/codes/open_flamingo/src/modeling_aki.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"fba7faec475b6155"}},{"code_sha256_prefix":"496711093e515803","entry":"DecoupledLinear","repo":"sony/aki","repo_kind":"official","path":"codes/open_flamingo/src/modeling_aki.py","file_url":"https://github.com/sony/aki/blob/HEAD/codes/open_flamingo/src/modeling_aki.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"496711093e515803"}},{"code_sha256_prefix":"0159490d2bc73030","entry":"PerceiverResampler","repo":"sony/aki","repo_kind":"official","path":"codes/open_flamingo/src/modeling_aki.py","file_url":"https://github.com/sony/aki/blob/HEAD/codes/open_flamingo/src/modeling_aki.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0159490d2bc73030"}},{"code_sha256_prefix":"fe68fff801021b23","entry":"getattr_recursive","repo":"sony/aki","repo_kind":"official","path":"codes/open_flamingo/src/modeling_aki.py","file_url":"https://github.com/sony/aki/blob/HEAD/codes/open_flamingo/src/modeling_aki.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"fe68fff801021b23"}},{"code_sha256_prefix":"660aaca8fc241af6","entry":"num_params","repo":"sony/aki","repo_kind":"official","path":"codes/open_flamingo/src/modeling_aki.py","file_url":"https://github.com/sony/aki/blob/HEAD/codes/open_flamingo/src/modeling_aki.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"660aaca8fc241af6"}},{"code_sha256_prefix":"5960fdbe295db3b4","entry":"stack_with_padding","repo":"sony/aki","repo_kind":"official","path":"codes/open_flamingo/src/modeling_aki.py","file_url":"https://github.com/sony/aki/blob/HEAD/codes/open_flamingo/src/modeling_aki.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5960fdbe295db3b4"}},{"code_sha256_prefix":"a2c6bf85eabc2a2d","entry":"AKI","repo":"sony/aki","repo_kind":"official","path":"codes/open_flamingo/src/modeling_aki.py","file_url":"https://github.com/sony/aki/blob/HEAD/codes/open_flamingo/src/modeling_aki.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a2c6bf85eabc2a2d"}},{"code_sha256_prefix":"54e0b46f8fc75e1f","entry":"VLM","repo":"sony/aki","repo_kind":"official","path":"codes/open_flamingo/src/modeling_aki.py","file_url":"https://github.com/sony/aki/blob/HEAD/codes/open_flamingo/src/modeling_aki.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"54e0b46f8fc75e1f"}},{"code_sha256_prefix":"d3d41f8bfc075e11","entry":"VLMOutputWithPast","repo":"sony/aki","repo_kind":"official","path":"codes/open_flamingo/src/modeling_aki.py","file_url":"https://github.com/sony/aki/blob/HEAD/codes/open_flamingo/src/modeling_aki.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d3d41f8bfc075e11"}},{"code_sha256_prefix":"05e29a0f3f27ca4d","entry":"VLMWithLanguageStream","repo":"sony/aki","repo_kind":"official","path":"codes/open_flamingo/src/modeling_aki.py","file_url":"https://github.com/sony/aki/blob/HEAD/codes/open_flamingo/src/modeling_aki.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"05e29a0f3f27ca4d"}},{"code_sha256_prefix":"710431cb836ef89c","entry":"VisionTokenizer","repo":"sony/aki","repo_kind":"official","path":"codes/open_flamingo/src/modeling_aki.py","file_url":"https://github.com/sony/aki/blob/HEAD/codes/open_flamingo/src/modeling_aki.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"710431cb836ef89c"}},{"code_sha256_prefix":"ca93e9c39ec815ad","entry":"stack_with_padding_2D_attention","repo":"sony/aki","repo_kind":"official","path":"codes/open_flamingo/src/modeling_aki.py","file_url":"https://github.com/sony/aki/blob/HEAD/codes/open_flamingo/src/modeling_aki.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ca93e9c39ec815ad"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}