{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/quadratic-is-not-what-you-need-for-multimodal","title":"Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to See","arxiv_id":"2410.06169","date":"2024-10-08","proceeding":null,"authors":["Zeliang Zhang","Phu Pham","Wentian Zhao","Kun Wan","Yu-Jhe Li","Jianing Zhou","Daniel Miranda","Ajinkya Kale","Chenliang Xu"],"abstract":"By treating visual tokens from visual encoders as text tokens, Multimodal Large Language Models (MLLMs) have achieved remarkable progress across diverse visual understanding tasks, leveraging the robust architectures of Large Language Models (LLMs). However, as token counts grow, the quadratic scaling of computation in LLMs introduces a significant efficiency bottleneck, impeding further scalability. Although recent approaches have explored pruning visual tokens or employing lighter LLM architectures, the computational overhead from an increasing number of visual tokens remains a substantial challenge. In this study, we investigate the redundancy in visual computation at both the parameter and computational pattern levels within LLaVA, a representative MLLM, and introduce a suite of streamlined strategies to enhance efficiency. These include neighbor-aware visual token attention, pruning of inactive visual attention heads, and selective layer dropping for visual computations. By implementing these strategies in LLaVA, we achieve a reduction in computational demands of 88% while maintaining model performance across key benchmarks. Additionally, we validate the existence of visual computational redundancy in other MLLMs, such as Qwen2-VL-7B and InternVL-2.0-4B/8B/26B. These results present a novel pathway for MLLMs to handle dense visual tokens with minimal computational costs. Code and model checkpoints will be released to support further research.","url_abs":"https://arxiv.org/abs/2410.06169v3","url_pdf":"https://arxiv.org/pdf/2410.06169v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"quadratic-is-not-what-you-need-for-multimodal","repo_url":"https://github.com/ZhangAIPI/YOPO_MLLM_Pruning","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"pruning","method_name":"Pruning"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.06169","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.06169"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ZhangAIPI/YOPO_MLLM_Pruning","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":2,"ran_draft_wrong":4,"ran_violates":1,"ran_honours":1,"unverified":2},"by_repo_kind":{"official":{"samples":10,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"ZhangAIPI/YOPO_MLLM_Pruning","repo_kind":"official","path":"modeling_llama_prune.py","file_url":"https://github.com/ZhangAIPI/YOPO_MLLM_Pruning/blob/HEAD/modeling_llama_prune.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"d61c483a3c2b3156","entry":"apply_rotary_pos_emb","repo":"ZhangAIPI/YOPO_MLLM_Pruning","repo_kind":"official","path":"modeling_llama_prune.py","file_url":"https://github.com/ZhangAIPI/YOPO_MLLM_Pruning/blob/HEAD/modeling_llama_prune.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d61c483a3c2b3156"}},{"code_sha256_prefix":"20e4f665698a3d18","entry":"collate_fn","repo":"ZhangAIPI/YOPO_MLLM_Pruning","repo_kind":"official","path":"LLaVA/llava/eval/model_vqa_loader.py","file_url":"https://github.com/ZhangAIPI/YOPO_MLLM_Pruning/blob/HEAD/LLaVA/llava/eval/model_vqa_loader.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"20e4f665698a3d18"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"ZhangAIPI/YOPO_MLLM_Pruning","repo_kind":"official","path":"LLaVA/llava/eval/model_vqa.py","file_url":"https://github.com/ZhangAIPI/YOPO_MLLM_Pruning/blob/HEAD/LLaVA/llava/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"bae18947b56f2be1","entry":"is_none","repo":"ZhangAIPI/YOPO_MLLM_Pruning","repo_kind":"official","path":"LLaVA/llava/eval/model_vqa_mmbench.py","file_url":"https://github.com/ZhangAIPI/YOPO_MLLM_Pruning/blob/HEAD/LLaVA/llava/eval/model_vqa_mmbench.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bae18947b56f2be1"}},{"code_sha256_prefix":"9b3c1cb391672ccb","entry":"load_image","repo":"ZhangAIPI/YOPO_MLLM_Pruning","repo_kind":"official","path":"LLaVA/predict.py","file_url":"https://github.com/ZhangAIPI/YOPO_MLLM_Pruning/blob/HEAD/LLaVA/predict.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9b3c1cb391672ccb"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"ZhangAIPI/YOPO_MLLM_Pruning","repo_kind":"official","path":"modeling_llama_prune.py","file_url":"https://github.com/ZhangAIPI/YOPO_MLLM_Pruning/blob/HEAD/modeling_llama_prune.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"ZhangAIPI/YOPO_MLLM_Pruning","repo_kind":"official","path":"LLaVA/llava/eval/model_vqa.py","file_url":"https://github.com/ZhangAIPI/YOPO_MLLM_Pruning/blob/HEAD/LLaVA/llava/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"eb4d16c3bba7cff8","entry":"apply_multimodal_rotary_pos_emb","repo":"ZhangAIPI/YOPO_MLLM_Pruning","repo_kind":"official","path":"modeling_qwen2_vl.py","file_url":"https://github.com/ZhangAIPI/YOPO_MLLM_Pruning/blob/HEAD/modeling_qwen2_vl.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eb4d16c3bba7cff8"}},{"code_sha256_prefix":"48f12d3a9c4a15e8","entry":"apply_rotary_pos_emb_vision","repo":"ZhangAIPI/YOPO_MLLM_Pruning","repo_kind":"official","path":"modeling_qwen2_vl.py","file_url":"https://github.com/ZhangAIPI/YOPO_MLLM_Pruning/blob/HEAD/modeling_qwen2_vl.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"48f12d3a9c4a15e8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}