{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/aim-adaptive-inference-of-multi-modal-llms","title":"AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruning","arxiv_id":"2412.03248","date":"2024-12-04","proceeding":null,"authors":["Yiwu Zhong","Zhuoming Liu","Yin Li","LiWei Wang"],"abstract":"Large language models (LLMs) have enabled the creation of multi-modal LLMs that exhibit strong comprehension of visual data such as images and videos. However, these models usually rely on extensive visual tokens from visual encoders, leading to high computational demands, which limits their applicability in resource-constrained environments and for long-context tasks. In this work, we propose a training-free adaptive inference method for multi-modal LLMs that can accommodate a broad range of efficiency requirements with a minimum performance drop. Our method consists of a) iterative token merging based on embedding similarity before LLMs, and b) progressive token pruning within LLM layers based on multi-modal importance. With a minimalist design, our method can be applied to both video and image LLMs. Extensive experiments on diverse video and image benchmarks demonstrate that, our method substantially reduces computation load (e.g., a $\\textbf{7-fold}$ reduction in FLOPs) while preserving the performance of video and image LLMs. Further, under a similar computational cost, our method outperforms the state-of-the-art methods in long video understanding (e.g., $\\textbf{+4.6}$ on MLVU). Additionally, our in-depth analysis provides insights into token redundancy and LLM layer behaviors, offering guidance for future research in designing efficient multi-modal LLMs. Our code will be available at https://github.com/LaVi-Lab/AIM.","url_abs":"https://arxiv.org/abs/2412.03248v1","url_pdf":"https://arxiv.org/pdf/2412.03248v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"aim-adaptive-inference-of-multi-modal-llms","repo_url":"https://github.com/lavi-lab/aim","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2412.03248","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.03248"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lavi-lab/aim","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":2,"ran_draft_wrong":2,"unverified":10},"by_repo_kind":{"official":{"samples":14,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b0a851cae92754ff","entry":"extract_patches","repo":"lavi-lab/aim","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/lavi-lab/aim/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b0a851cae92754ff"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"lavi-lab/aim","repo_kind":"official","path":"llava/eval/model_vqa.py","file_url":"https://github.com/lavi-lab/aim/blob/HEAD/llava/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"lavi-lab/aim","repo_kind":"official","path":"llava/eval/model_vqa.py","file_url":"https://github.com/lavi-lab/aim/blob/HEAD/llava/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"7606525af238fb64","entry":"unpad_image","repo":"lavi-lab/aim","repo_kind":"official","path":"llava/model/llava_arch.py","file_url":"https://github.com/lavi-lab/aim/blob/HEAD/llava/model/llava_arch.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7606525af238fb64"}},{"code_sha256_prefix":"3c8823948724e269","entry":"auto_pad_images","repo":"lavi-lab/aim","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/lavi-lab/aim/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3c8823948724e269"}},{"code_sha256_prefix":"7d78565ff76da4dc","entry":"bipartite_soft_matching_merge","repo":"lavi-lab/aim","repo_kind":"official","path":"llava/model/token_merge.py","file_url":"https://github.com/lavi-lab/aim/blob/HEAD/llava/model/token_merge.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7d78565ff76da4dc"}},{"code_sha256_prefix":"c32b636f13ed9ee5","entry":"convert_to_scalar","repo":"lavi-lab/aim","repo_kind":"official","path":"trl/core.py","file_url":"https://github.com/lavi-lab/aim/blob/HEAD/trl/core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c32b636f13ed9ee5"}},{"code_sha256_prefix":"a51ae45a1f6c0b5b","entry":"flatten_dict","repo":"lavi-lab/aim","repo_kind":"official","path":"trl/core.py","file_url":"https://github.com/lavi-lab/aim/blob/HEAD/trl/core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a51ae45a1f6c0b5b"}},{"code_sha256_prefix":"837152d923b7a4d0","entry":"merging_ops","repo":"lavi-lab/aim","repo_kind":"official","path":"other_packages/LLM-Viewer/analyze_aim.py","file_url":"https://github.com/lavi-lab/aim/blob/HEAD/other_packages/LLM-Viewer/analyze_aim.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"837152d923b7a4d0"}},{"code_sha256_prefix":"577d19646599cf85","entry":"merging_ops_iter","repo":"lavi-lab/aim","repo_kind":"official","path":"other_packages/LLM-Viewer/analyze_aim.py","file_url":"https://github.com/lavi-lab/aim/blob/HEAD/other_packages/LLM-Viewer/analyze_aim.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"577d19646599cf85"}},{"code_sha256_prefix":"edd36a84c6f8d46e","entry":"pruning_ops_layer","repo":"lavi-lab/aim","repo_kind":"official","path":"other_packages/LLM-Viewer/analyze_aim.py","file_url":"https://github.com/lavi-lab/aim/blob/HEAD/other_packages/LLM-Viewer/analyze_aim.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"edd36a84c6f8d46e"}},{"code_sha256_prefix":"c80358dff96c25ce","entry":"resize_and_center_crop","repo":"lavi-lab/aim","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/lavi-lab/aim/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c80358dff96c25ce"}},{"code_sha256_prefix":"b20705b0dfd77ffc","entry":"safe_load_tokenizer","repo":"lavi-lab/aim","repo_kind":"official","path":"llava/conversation.py","file_url":"https://github.com/lavi-lab/aim/blob/HEAD/llava/conversation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b20705b0dfd77ffc"}},{"code_sha256_prefix":"2c3da386578af970","entry":"top_k_top_p_filtering","repo":"lavi-lab/aim","repo_kind":"official","path":"trl/core.py","file_url":"https://github.com/lavi-lab/aim/blob/HEAD/trl/core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2c3da386578af970"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}