{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dynamic-llava-efficient-multimodal-large","title":"Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification","arxiv_id":"2412.00876","date":"2024-12-01","proceeding":null,"authors":["Wenxuan Huang","Zijie Zhai","Yunhang Shen","Shaosheng Cao","Fei Zhao","Xiangfeng Xu","Zheyu Ye","Shaohui Lin"],"abstract":"Multimodal Large Language Models (MLLMs) have achieved remarkable success in vision understanding, reasoning, and interaction. However, the inference computation and memory increase progressively with the generation of output tokens during decoding, directly affecting the efficacy of MLLMs. Existing methods attempt to reduce the vision context redundancy to achieve efficient MLLMs. Unfortunately, the efficiency benefits of the vision context reduction in the prefill stage gradually diminish during the decoding stage. To address this problem, we proposed a dynamic vision-language context sparsification framework Dynamic-LLaVA, which dynamically reduces the redundancy of vision context in the prefill stage and decreases the memory and computation overhead of the generated language context during decoding. Dynamic-LLaVA designs a tailored sparsification inference scheme for different inference modes, i.e., prefill, decoding with and without KV cache, to achieve efficient inference of MLLMs. In practice, Dynamic-LLaVA can reduce computation consumption by $\\sim$75\\% in the prefill stage. Meanwhile, throughout the entire generation process of MLLMs, Dynamic-LLaVA reduces the $\\sim$50\\% computation consumption under decoding without KV cache, while saving $\\sim$50\\% GPU memory overhead when decoding with KV cache, due to the vision-language context sparsification. Extensive experiments also demonstrate that Dynamic-LLaVA achieves efficient inference for MLLMs with negligible understanding and generation ability degradation or even performance gains compared to the full-context inference baselines. Code is available at https://github.com/Osilly/dynamic_llava .","url_abs":"https://arxiv.org/abs/2412.00876v3","url_pdf":"https://arxiv.org/pdf/2412.00876v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dynamic-llava-efficient-multimodal-large","repo_url":"https://github.com/osilly/dynamic_llava","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-question-answering-on-mm-vet","task":"Visual Question Answering","dataset":"MM-Vet","model":"Dynamic-LLaVA-13B","rank_in_archive_order":135,"of":231,"metrics":{"GPT-4 score":"37.3"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-mm-vet","task":"Visual Question Answering","dataset":"MM-Vet","model":"Dynamic-LLaVA-7B","rank_in_archive_order":184,"of":231,"metrics":{"GPT-4 score":"32.2"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.00876","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.00876"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Osilly/dynamic_llava","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/osilly/dynamic_llava","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4,"ran_draft_wrong":7,"ran_violates":1,"ran_fixture":1,"unverified":5},"by_repo_kind":{"official":{"samples":18,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"68bd42b9c09a9dd1","entry":"LlamaDynamicNTKScalingRotaryEmbedding","repo":"osilly/dynamic_llava","repo_kind":"official","path":"llava/model/language_model/dynamic_modeling_llama.py","file_url":"https://github.com/osilly/dynamic_llava/blob/HEAD/llava/model/language_model/dynamic_modeling_llama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"68bd42b9c09a9dd1"}},{"code_sha256_prefix":"1287c98591685f54","entry":"LlamaLinearScalingRotaryEmbedding","repo":"osilly/dynamic_llava","repo_kind":"official","path":"llava/model/language_model/dynamic_modeling_llama.py","file_url":"https://github.com/osilly/dynamic_llava/blob/HEAD/llava/model/language_model/dynamic_modeling_llama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1287c98591685f54"}},{"code_sha256_prefix":"c5d348f0b0059642","entry":"LlamaMLP","repo":"osilly/dynamic_llava","repo_kind":"official","path":"llava/model/language_model/dynamic_modeling_llama.py","file_url":"https://github.com/osilly/dynamic_llava/blob/HEAD/llava/model/language_model/dynamic_modeling_llama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c5d348f0b0059642"}},{"code_sha256_prefix":"7541f97d86797bda","entry":"LlamaRotaryEmbedding","repo":"osilly/dynamic_llava","repo_kind":"official","path":"llava/model/language_model/dynamic_modeling_llama.py","file_url":"https://github.com/osilly/dynamic_llava/blob/HEAD/llava/model/language_model/dynamic_modeling_llama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7541f97d86797bda"}},{"code_sha256_prefix":"20e4f665698a3d18","entry":"collate_fn","repo":"Osilly/dynamic_llava","repo_kind":"official","path":"llava/dynamic_eval/model_vqa_loader.py","file_url":"https://github.com/Osilly/dynamic_llava/blob/HEAD/llava/dynamic_eval/model_vqa_loader.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"20e4f665698a3d18"}},{"code_sha256_prefix":"94fda0761ce0d5b9","entry":"convert_caps","repo":"Osilly/dynamic_llava","repo_kind":"official","path":"llava/dynamic_eval/eval_science_qa.py","file_url":"https://github.com/Osilly/dynamic_llava/blob/HEAD/llava/dynamic_eval/eval_science_qa.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"94fda0761ce0d5b9"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"Osilly/dynamic_llava","repo_kind":"official","path":"llava/dynamic_eval/model_lvis_for_meteor.py","file_url":"https://github.com/Osilly/dynamic_llava/blob/HEAD/llava/dynamic_eval/model_lvis_for_meteor.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"36e875f85baa4565","entry":"get_pred_idx","repo":"Osilly/dynamic_llava","repo_kind":"official","path":"llava/dynamic_eval/eval_science_qa.py","file_url":"https://github.com/Osilly/dynamic_llava/blob/HEAD/llava/dynamic_eval/eval_science_qa.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"36e875f85baa4565"}},{"code_sha256_prefix":"1e68840d6bbdface","entry":"prompt_processor","repo":"Osilly/dynamic_llava","repo_kind":"official","path":"llava/dynamic_eval/eval_textvqa.py","file_url":"https://github.com/Osilly/dynamic_llava/blob/HEAD/llava/dynamic_eval/eval_textvqa.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1e68840d6bbdface"}},{"code_sha256_prefix":"bc218bfbe73e286b","entry":"scaled_dot_product_attention_with_policy","repo":"osilly/dynamic_llava","repo_kind":"official","path":"llava/model/language_model/dynamic_modeling_llama.py","file_url":"https://github.com/osilly/dynamic_llava/blob/HEAD/llava/model/language_model/dynamic_modeling_llama.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bc218bfbe73e286b"}},{"code_sha256_prefix":"aa7e8923d37f93a8","entry":"softmax_with_policy","repo":"osilly/dynamic_llava","repo_kind":"official","path":"llava/model/language_model/dynamic_modeling_llama.py","file_url":"https://github.com/osilly/dynamic_llava/blob/HEAD/llava/model/language_model/dynamic_modeling_llama.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aa7e8923d37f93a8"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"Osilly/dynamic_llava","repo_kind":"official","path":"llava/dynamic_eval/model_lvis_for_meteor.py","file_url":"https://github.com/Osilly/dynamic_llava/blob/HEAD/llava/dynamic_eval/model_lvis_for_meteor.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"55c32993da87759b","entry":"unpad_image","repo":"Osilly/dynamic_llava","repo_kind":"official","path":"llava/model/dynamic_llava_arch.py","file_url":"https://github.com/Osilly/dynamic_llava/blob/HEAD/llava/model/dynamic_llava_arch.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"55c32993da87759b"}},{"code_sha256_prefix":"d866787c2702613f","entry":"Cache","repo":"osilly/dynamic_llava","repo_kind":"official","path":"llava/model/language_model/dynamic_modeling_llama.py","file_url":"https://github.com/osilly/dynamic_llava/blob/HEAD/llava/model/language_model/dynamic_modeling_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d866787c2702613f"}},{"code_sha256_prefix":"2ee5fbf088ba5b7e","entry":"DynamicLlamaDecoderLayer","repo":"osilly/dynamic_llava","repo_kind":"official","path":"llava/model/language_model/dynamic_modeling_llama.py","file_url":"https://github.com/osilly/dynamic_llava/blob/HEAD/llava/model/language_model/dynamic_modeling_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2ee5fbf088ba5b7e"}},{"code_sha256_prefix":"c0464d1e0119ec7f","entry":"DynamicLlamaSdpaAttention","repo":"osilly/dynamic_llava","repo_kind":"official","path":"llava/model/language_model/dynamic_modeling_llama.py","file_url":"https://github.com/osilly/dynamic_llava/blob/HEAD/llava/model/language_model/dynamic_modeling_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c0464d1e0119ec7f"}},{"code_sha256_prefix":"64c5b45fc73cecfd","entry":"LlamaAttention","repo":"osilly/dynamic_llava","repo_kind":"official","path":"llava/model/language_model/dynamic_modeling_llama.py","file_url":"https://github.com/osilly/dynamic_llava/blob/HEAD/llava/model/language_model/dynamic_modeling_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"64c5b45fc73cecfd"}},{"code_sha256_prefix":"58c176deebe204d2","entry":"LlamaFlashAttention2","repo":"osilly/dynamic_llava","repo_kind":"official","path":"llava/model/language_model/dynamic_modeling_llama.py","file_url":"https://github.com/osilly/dynamic_llava/blob/HEAD/llava/model/language_model/dynamic_modeling_llama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"58c176deebe204d2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}