{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-generalist-robot-policies-what","title":"Towards Generalist Robot Policies: What Matters in Building Vision-Language-Action Models","arxiv_id":"2412.14058","date":"2024-12-18","proceeding":null,"authors":["Xinghang Li","Peiyan Li","Minghuan Liu","Dong Wang","Jirong Liu","Bingyi Kang","Xiao Ma","Tao Kong","Hanbo Zhang","Huaping Liu"],"abstract":"Foundation Vision Language Models (VLMs) exhibit strong capabilities in multi-modal representation learning, comprehension, and reasoning. By injecting action components into the VLMs, Vision-Language-Action Models (VLAs) can be naturally formed and also show promising performance. Existing work has demonstrated the effectiveness and generalization of VLAs in multiple scenarios and tasks. Nevertheless, the transfer from VLMs to VLAs is not trivial since existing VLAs differ in their backbones, action-prediction formulations, data distributions, and training recipes. This leads to a missing piece for a systematic understanding of the design choices of VLAs. In this work, we disclose the key factors that significantly influence the performance of VLA and focus on answering three essential design choices: which backbone to select, how to formulate the VLA architectures, and when to add cross-embodiment data. The obtained results convince us firmly to explain why we need VLA and develop a new family of VLAs, RoboVLMs, which require very few manual designs and achieve a new state-of-the-art performance in three simulation tasks and real-world experiments. Through our extensive experiments, which include over 8 VLM backbones, 4 policy architectures, and over 600 distinct designed experiments, we provide a detailed guidebook for the future design of VLAs. In addition to the study, the highly flexible RoboVLMs framework, which supports easy integrations of new VLMs and free combinations of various design choices, is made public to facilitate future research. We open-source all details, including codes, models, datasets, and toolkits, along with detailed training and evaluation recipes at: robovlms.github.io.","url_abs":"https://arxiv.org/abs/2412.14058v3","url_pdf":"https://arxiv.org/pdf/2412.14058v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-generalist-robot-policies-what","repo_url":"https://github.com/Robot-VLAs/RoboVLMs","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"robot-manipulation","task_name":"Robot Manipulation"},{"task_slug":"vision-language-action","task_name":"Vision-Language-Action"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/robot-manipulation-on-calvin","task":"Robot Manipulation","dataset":"CALVIN","model":"RoboVLMs","rank_in_archive_order":3,"of":19,"metrics":{"avg. sequence length (D to D)":"4.25"},"uses_additional_data":false},{"leaderboard":"/sota/robot-manipulation-on-simpler-env","task":"Robot Manipulation","dataset":"SimplerEnv-Google Robot","model":"RoboVLM","rank_in_archive_order":5,"of":9,"metrics":{"Variant Aggregation":"0.463","Variant Aggregation-Move Near":"0.560","Variant Aggregation-Open/Close Drawer":"0.085","Variant Aggregation-Pick Coke Can":"0.683","Visual Matching":"0.563","Visual Matching-Move Near":"0.663","Visual Matching-Open/Close Drawer":"0.268","Visual Matching-Pick Coke Can":"0.727"},"uses_additional_data":true},{"leaderboard":"/sota/robot-manipulation-on-simplerenv-widow-x","task":"Robot Manipulation","dataset":"SimplerEnv-Widow X","model":"RoboVLM","rank_in_archive_order":5,"of":7,"metrics":{"Average":"0.135","Put Carrot on Plate":"0.250","Put Eggplant  in Yellow Basket":"0.000","Put Spoon on Towel":"0.208","Stack Green Block on Yellow Block":"0.083"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.14058","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.14058"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Robot-VLAs/RoboVLMs","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_honours":1,"unverified":7},"by_repo_kind":{"listed":{"samples":8,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3cab76f0744f529e","entry":"adjust_learning_rate","repo":"Robot-VLAs/RoboVLMs","repo_kind":"listed","path":"robovlms/utils/model_utils.py","file_url":"https://github.com/Robot-VLAs/RoboVLMs/blob/HEAD/robovlms/utils/model_utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3cab76f0744f529e"}},{"code_sha256_prefix":"2d13f18ccbf39778","entry":"build_llm_flamingo","repo":"Robot-VLAs/RoboVLMs","repo_kind":"listed","path":"robovlms/model/flamingo_builder.py","file_url":"https://github.com/Robot-VLAs/RoboVLMs/blob/HEAD/robovlms/model/flamingo_builder.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2d13f18ccbf39778"}},{"code_sha256_prefix":"a3053ba311231c5a","entry":"calculate_vl_cross_entropy","repo":"Robot-VLAs/RoboVLMs","repo_kind":"listed","path":"robovlms/train/loss.py","file_url":"https://github.com/Robot-VLAs/RoboVLMs/blob/HEAD/robovlms/train/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a3053ba311231c5a"}},{"code_sha256_prefix":"5cf661f640d4a718","entry":"deep_update","repo":"Robot-VLAs/RoboVLMs","repo_kind":"listed","path":"robovlms/model/backbone/base_backbone.py","file_url":"https://github.com/Robot-VLAs/RoboVLMs/blob/HEAD/robovlms/model/backbone/base_backbone.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5cf661f640d4a718"}},{"code_sha256_prefix":"9e1210cd3b842e0c","entry":"default_llm_flamingo_config","repo":"Robot-VLAs/RoboVLMs","repo_kind":"listed","path":"robovlms/model/flamingo_builder.py","file_url":"https://github.com/Robot-VLAs/RoboVLMs/blob/HEAD/robovlms/model/flamingo_builder.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9e1210cd3b842e0c"}},{"code_sha256_prefix":"649d5af3e1cc5444","entry":"default_tokenizer_config","repo":"Robot-VLAs/RoboVLMs","repo_kind":"listed","path":"robovlms/utils/model_utils.py","file_url":"https://github.com/Robot-VLAs/RoboVLMs/blob/HEAD/robovlms/utils/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"649d5af3e1cc5444"}},{"code_sha256_prefix":"c6f0632932821291","entry":"load_config","repo":"Robot-VLAs/RoboVLMs","repo_kind":"listed","path":"robovlms/model/backbone/base_backbone.py","file_url":"https://github.com/Robot-VLAs/RoboVLMs/blob/HEAD/robovlms/model/backbone/base_backbone.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c6f0632932821291"}},{"code_sha256_prefix":"170e61058de0df72","entry":"update_tokenizer","repo":"Robot-VLAs/RoboVLMs","repo_kind":"listed","path":"robovlms/utils/model_utils.py","file_url":"https://github.com/Robot-VLAs/RoboVLMs/blob/HEAD/robovlms/utils/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"170e61058de0df72"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}