{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deer-vla-dynamic-inference-of-multimodal","title":"DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution","arxiv_id":"2411.02359","date":"2024-11-04","proceeding":null,"authors":["Yang Yue","Yulin Wang","Bingyi Kang","Yizeng Han","Shenzhi Wang","Shiji Song","Jiashi Feng","Gao Huang"],"abstract":"MLLMs have demonstrated remarkable comprehension and reasoning capabilities with complex language and visual data. These advances have spurred the vision of establishing a generalist robotic MLLM proficient in understanding complex human instructions and accomplishing various embodied tasks. However, developing MLLMs for real-world robots is challenging due to the typically limited computation and memory capacities available on robotic platforms. In contrast, the inference of MLLMs involves storing billions of parameters and performing tremendous computation, imposing significant hardware demands. In our paper, we propose a Dynamic Early-Exit Framework for Robotic Vision-Language-Action Model (DeeR-VLA, or simply DeeR) that automatically adjusts the size of the activated MLLM based on each situation at hand. The approach leverages a multi-exit architecture in MLLMs, which allows the model to terminate processing once a proper size of the model has been activated for a specific situation, thus avoiding further redundant computation. Additionally, we develop novel algorithms that establish early-termination criteria for DeeR, conditioned on predefined demands such as average computational cost (i.e., power consumption), as well as peak computational consumption (i.e., latency) and GPU memory usage. These enhancements ensure that DeeR operates efficiently under varying resource constraints while maintaining competitive performance. On the CALVIN robot manipulation benchmark, DeeR demonstrates significant reductions in computational costs of LLM by 5.2-6.5x and GPU memory of LLM by 2-6x without compromising performance. Code and checkpoints are available at https://github.com/yueyang130/DeeR-VLA.","url_abs":"https://arxiv.org/abs/2411.02359v1","url_pdf":"https://arxiv.org/pdf/2411.02359v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deer-vla-dynamic-inference-of-multimodal","repo_url":"https://github.com/yueyang130/deer-vla","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"robot-manipulation","task_name":"Robot Manipulation"},{"task_slug":"vision-language-action","task_name":"Vision-Language-Action"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2411.02359","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.02359"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/yueyang130/DeeR-VLA","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yueyang130/deer-vla","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":10},"by_repo_kind":{"official":{"samples":10,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ad9e1b77ed2429b2","entry":"apply_conditioning","repo":"yueyang130/DeeR-VLA","repo_kind":"official","path":"robot_flamingo/models/unets.py","file_url":"https://github.com/yueyang130/DeeR-VLA/blob/HEAD/robot_flamingo/models/unets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ad9e1b77ed2429b2"}},{"code_sha256_prefix":"b5f436d39b3d4b3f","entry":"cosine_beta_schedule","repo":"yueyang130/DeeR-VLA","repo_kind":"official","path":"robot_flamingo/models/unets.py","file_url":"https://github.com/yueyang130/DeeR-VLA/blob/HEAD/robot_flamingo/models/unets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b5f436d39b3d4b3f"}},{"code_sha256_prefix":"7bf6ead301088975","entry":"dict_apply","repo":"yueyang130/DeeR-VLA","repo_kind":"official","path":"robot_flamingo/models/normalizer.py","file_url":"https://github.com/yueyang130/DeeR-VLA/blob/HEAD/robot_flamingo/models/normalizer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7bf6ead301088975"}},{"code_sha256_prefix":"09c8479d9a5b3e06","entry":"extract","repo":"yueyang130/DeeR-VLA","repo_kind":"official","path":"robot_flamingo/models/unets.py","file_url":"https://github.com/yueyang130/DeeR-VLA/blob/HEAD/robot_flamingo/models/unets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"09c8479d9a5b3e06"}},{"code_sha256_prefix":"52bc358247535eaf","entry":"gen_attention_mask_in_length","repo":"yueyang130/DeeR-VLA","repo_kind":"official","path":"modeling_gpt_9b.py","file_url":"https://github.com/yueyang130/DeeR-VLA/blob/HEAD/modeling_gpt_9b.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"52bc358247535eaf"}},{"code_sha256_prefix":"d3c6b00a69aea30c","entry":"gen_flash_attn_padding_info","repo":"yueyang130/DeeR-VLA","repo_kind":"official","path":"modeling_gpt_9b.py","file_url":"https://github.com/yueyang130/DeeR-VLA/blob/HEAD/modeling_gpt_9b.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d3c6b00a69aea30c"}},{"code_sha256_prefix":"e28872db9ca12819","entry":"get_observation","repo":"yueyang130/DeeR-VLA","repo_kind":"official","path":"bayesian_optimization.py","file_url":"https://github.com/yueyang130/DeeR-VLA/blob/HEAD/bayesian_optimization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e28872db9ca12819"}},{"code_sha256_prefix":"1fff25335fef27a2","entry":"get_score","repo":"yueyang130/DeeR-VLA","repo_kind":"official","path":"bayesian_optimization.py","file_url":"https://github.com/yueyang130/DeeR-VLA/blob/HEAD/bayesian_optimization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1fff25335fef27a2"}},{"code_sha256_prefix":"00a33466c69c5705","entry":"load_tf_weights_in_gpt2","repo":"yueyang130/DeeR-VLA","repo_kind":"official","path":"robot_flamingo/models/trajectory_gpt2.py","file_url":"https://github.com/yueyang130/DeeR-VLA/blob/HEAD/robot_flamingo/models/trajectory_gpt2.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"00a33466c69c5705"}},{"code_sha256_prefix":"8717483e23e5393b","entry":"lstm_decoder","repo":"yueyang130/DeeR-VLA","repo_kind":"official","path":"robot_flamingo/models/action_head.py","file_url":"https://github.com/yueyang130/DeeR-VLA/blob/HEAD/robot_flamingo/models/action_head.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8717483e23e5393b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}