{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/look-m-look-once-optimization-in-kv-cache-for","title":"LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference","arxiv_id":"2406.18139","date":"2024-06-26","proceeding":null,"authors":["Zhongwei Wan","Ziang Wu","Che Liu","Jinfa Huang","Zhihong Zhu","Peng Jin","Longyue Wang","Li Yuan"],"abstract":"Long-context Multimodal Large Language Models (MLLMs) demand substantial computational resources for inference as the growth of their multimodal Key-Value (KV) cache, in response to increasing input lengths, challenges memory and time efficiency. Unlike single-modality LLMs that manage only textual contexts, the KV cache of long-context MLLMs includes representations from multiple images with temporal and spatial relationships and related textual contexts. The predominance of image tokens means traditional optimizations for LLMs' KV caches are unsuitable for multimodal long-context settings, and no prior works have addressed this challenge. In this work, we introduce LOOK-M, a pioneering, fine-tuning-free approach that efficiently reduces the multimodal KV cache size while maintaining performance comparable to a full cache. We observe that during prompt prefill, the model prioritizes more textual attention over image features, and based on the multimodal interaction observation, a new proposed text-prior method is explored to compress the KV cache. Furthermore, to mitigate the degradation of image contextual information, we propose several compensatory strategies using KV pairs merging. LOOK-M demonstrates that with a significant reduction in KV Cache memory usage, such as reducing it by 80% in some cases, it not only achieves up to 1.5x faster decoding but also maintains or even enhances performance across a variety of long context multimodal tasks.","url_abs":"https://arxiv.org/abs/2406.18139v1","url_pdf":"https://arxiv.org/pdf/2406.18139v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"look-m-look-once-optimization-in-kv-cache-for","repo_url":"https://github.com/sustechbruce/look-m","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"multimodal-interaction","task_name":"multimodal interaction"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.18139","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.18139"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sustechbruce/look-m","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":2,"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"184a512ccc58837f","entry":"dict_mean","repo":"sustechbruce/look-m","repo_kind":"official","path":"score.py","file_url":"https://github.com/sustechbruce/look-m/blob/HEAD/score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"184a512ccc58837f"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"sustechbruce/look-m","repo_kind":"official","path":"LLaVA-mix_merge_v1/llava/eval/model_vqa.py","file_url":"https://github.com/sustechbruce/look-m/blob/HEAD/LLaVA-mix_merge_v1/llava/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"1d12e02f36106cf1","entry":"split_data","repo":"sustechbruce/look-m","repo_kind":"official","path":"generate.py","file_url":"https://github.com/sustechbruce/look-m/blob/HEAD/generate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1d12e02f36106cf1"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"sustechbruce/look-m","repo_kind":"official","path":"LLaVA-mix_merge_v1/llava/eval/model_vqa.py","file_url":"https://github.com/sustechbruce/look-m/blob/HEAD/LLaVA-mix_merge_v1/llava/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"d4e3113001afa6cb","entry":"main","repo":"sustechbruce/look-m","repo_kind":"official","path":"score.py","file_url":"https://github.com/sustechbruce/look-m/blob/HEAD/score.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d4e3113001afa6cb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}