{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cross-modal-information-flow-in-multimodal","title":"Cross-modal Information Flow in Multimodal Large Language Models","arxiv_id":"2411.18620","date":"2024-11-27","proceeding":"CVPR 2025 1","authors":["Zhi Zhang","Srishti Yadav","Fengze Han","Ekaterina Shutova"],"abstract":"The recent advancements in auto-regressive multimodal large language models (MLLMs) have demonstrated promising progress for vision-language tasks. While there exists a variety of studies investigating the processing of linguistic information within large language models, little is currently known about the inner working mechanism of MLLMs and how linguistic and visual information interact within these models. In this study, we aim to fill this gap by examining the information flow between different modalities -- language and vision -- in MLLMs, focusing on visual question answering. Specifically, given an image-question pair as input, we investigate where in the model and how the visual and linguistic information are combined to generate the final prediction. Conducting experiments with a series of models from the LLaVA series, we find that there are two distinct stages in the process of integration of the two modalities. In the lower layers, the model first transfers the more general visual features of the whole image into the representations of (linguistic) question tokens. In the middle layers, it once again transfers visual information about specific objects relevant to the question to the respective token positions of the question. Finally, in the higher layers, the resulting multimodal representation is propagated to the last position of the input sequence for the final prediction. Overall, our findings provide a new and comprehensive perspective on the spatial and functional aspects of image and language processing in the MLLMs, thereby facilitating future research into multimodal information localization and editing. Our code and collected dataset are released here: https://github.com/FightingFighting/cross-modal-information-flow-in-MLLM.git.","url_abs":"https://arxiv.org/abs/2411.18620v2","url_pdf":"https://arxiv.org/pdf/2411.18620v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cross-modal-information-flow-in-multimodal","repo_url":"https://github.com/FightingFighting/cross-modal-information-flow-in-MLLM","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.18620","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.18620"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/FightingFighting/cross-modal-information-flow-in-MLLM","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":11},"by_repo_kind":{"official":{"samples":11,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b3e90da48b755b82","entry":"cache_hiddenstate","repo":"FightingFighting/cross-modal-information-flow-in-MLLM","repo_kind":"official","path":"last_position_answer_prob.py","file_url":"https://github.com/FightingFighting/cross-modal-information-flow-in-MLLM/blob/HEAD/last_position_answer_prob.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b3e90da48b755b82"}},{"code_sha256_prefix":"bd5d01a52aacfde0","entry":"collate_fn","repo":"FightingFighting/cross-modal-information-flow-in-MLLM","repo_kind":"official","path":"InformationFlow.py","file_url":"https://github.com/FightingFighting/cross-modal-information-flow-in-MLLM/blob/HEAD/InformationFlow.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd5d01a52aacfde0"}},{"code_sha256_prefix":"9532c9c8f3eec45b","entry":"decode_tokens","repo":"FightingFighting/cross-modal-information-flow-in-MLLM","repo_kind":"official","path":"utils.py","file_url":"https://github.com/FightingFighting/cross-modal-information-flow-in-MLLM/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9532c9c8f3eec45b"}},{"code_sha256_prefix":"face49680c2f6521","entry":"find_token_range","repo":"FightingFighting/cross-modal-information-flow-in-MLLM","repo_kind":"official","path":"InformationFlow.py","file_url":"https://github.com/FightingFighting/cross-modal-information-flow-in-MLLM/blob/HEAD/InformationFlow.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"face49680c2f6521"}},{"code_sha256_prefix":"c82183ea3f76d9b0","entry":"find_token_range","repo":"FightingFighting/cross-modal-information-flow-in-MLLM","repo_kind":"official","path":"utils.py","file_url":"https://github.com/FightingFighting/cross-modal-information-flow-in-MLLM/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c82183ea3f76d9b0"}},{"code_sha256_prefix":"efd2dff547322153","entry":"make_inputs","repo":"FightingFighting/cross-modal-information-flow-in-MLLM","repo_kind":"official","path":"utils.py","file_url":"https://github.com/FightingFighting/cross-modal-information-flow-in-MLLM/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"efd2dff547322153"}},{"code_sha256_prefix":"862b3c59f9afa821","entry":"read_csv","repo":"FightingFighting/cross-modal-information-flow-in-MLLM","repo_kind":"official","path":"vil/merge_lineplot.py","file_url":"https://github.com/FightingFighting/cross-modal-information-flow-in-MLLM/blob/HEAD/vil/merge_lineplot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"862b3c59f9afa821"}},{"code_sha256_prefix":"70e0ea54db616e76","entry":"run_original","repo":"FightingFighting/cross-modal-information-flow-in-MLLM","repo_kind":"official","path":"last_position_answer_prob.py","file_url":"https://github.com/FightingFighting/cross-modal-information-flow-in-MLLM/blob/HEAD/last_position_answer_prob.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"70e0ea54db616e76"}},{"code_sha256_prefix":"4af864b9e3abfdf0","entry":"set_block_attn_add_hooks_llava","repo":"FightingFighting/cross-modal-information-flow-in-MLLM","repo_kind":"official","path":"methods.py","file_url":"https://github.com/FightingFighting/cross-modal-information-flow-in-MLLM/blob/HEAD/methods.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4af864b9e3abfdf0"}},{"code_sha256_prefix":"b0b63267dec9b012","entry":"set_block_attn_hooks_llava","repo":"FightingFighting/cross-modal-information-flow-in-MLLM","repo_kind":"official","path":"methods.py","file_url":"https://github.com/FightingFighting/cross-modal-information-flow-in-MLLM/blob/HEAD/methods.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b0b63267dec9b012"}},{"code_sha256_prefix":"0f3a60de73d84bd9","entry":"set_get_attn_proj_hooks","repo":"FightingFighting/cross-modal-information-flow-in-MLLM","repo_kind":"official","path":"methods.py","file_url":"https://github.com/FightingFighting/cross-modal-information-flow-in-MLLM/blob/HEAD/methods.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f3a60de73d84bd9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}