{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pointvla-injecting-the-3d-world-into-vision","title":"PointVLA: Injecting the 3D World into Vision-Language-Action Models","arxiv_id":"2503.07511","date":"2025-03-10","proceeding":null,"authors":["Chengmeng Li","Junjie Wen","Yan Peng","Yaxin Peng","Feifei Feng","Yichen Zhu"],"abstract":"Vision-Language-Action (VLA) models excel at robotic tasks by leveraging large-scale 2D vision-language pretraining, but their reliance on RGB images limits spatial reasoning critical for real-world interaction. Retraining these models with 3D data is computationally prohibitive, while discarding existing 2D datasets wastes valuable resources. To bridge this gap, we propose PointVLA, a framework that enhances pre-trained VLAs with point cloud inputs without requiring retraining. Our method freezes the vanilla action expert and injects 3D features via a lightweight modular block. To identify the most effective way of integrating point cloud representations, we conduct a skip-block analysis to pinpoint less useful blocks in the vanilla action expert, ensuring that 3D features are injected only into these blocks--minimizing disruption to pre-trained representations. Extensive experiments demonstrate that PointVLA outperforms state-of-the-art 2D imitation learning methods, such as OpenVLA, Diffusion Policy and DexVLA, across both simulated and real-world robotic tasks. Specifically, we highlight several key advantages of PointVLA enabled by point cloud integration: (1) Few-shot multi-tasking, where PointVLA successfully performs four different tasks using only 20 demonstrations each; (2) Real-vs-photo discrimination, where PointVLA distinguishes real objects from their images, leveraging 3D world knowledge to improve safety and reliability; (3) Height adaptability, Unlike conventional 2D imitation learning methods, PointVLA enables robots to adapt to objects at varying table height that unseen in train data. Furthermore, PointVLA achieves strong performance in long-horizon tasks, such as picking and packing objects from a moving conveyor belt, showcasing its ability to generalize across complex, dynamic environments.","url_abs":"https://arxiv.org/abs/2503.07511v1","url_pdf":"https://arxiv.org/pdf/2503.07511v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pointvla-injecting-the-3d-world-into-vision","repo_url":"https://github.com/YanjieZe/Improved-3D-Diffusion-Policy","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"imitation-learning","task_name":"Imitation Learning"},{"task_slug":"spatial-reasoning","task_name":"Spatial Reasoning"},{"task_slug":"vision-language-action","task_name":"Vision-Language-Action"},{"task_slug":"world-knowledge","task_name":"World Knowledge"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.07511","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.07511"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/YanjieZe/Improved-3D-Diffusion-Policy","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"unverified":8},"by_repo_kind":{"listed":{"samples":15,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0a16eab1dfc7ced0","entry":"create_indices","repo":"YanjieZe/Improved-3D-Diffusion-Policy","repo_kind":"listed","path":"Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/sampler.py","file_url":"https://github.com/YanjieZe/Improved-3D-Diffusion-Policy/blob/HEAD/Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/sampler.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a16eab1dfc7ced0"}},{"code_sha256_prefix":"e889bd81b4735ee5","entry":"dict_apply","repo":"YanjieZe/Improved-3D-Diffusion-Policy","repo_kind":"listed","path":"Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/pytorch_util.py","file_url":"https://github.com/YanjieZe/Improved-3D-Diffusion-Policy/blob/HEAD/Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/pytorch_util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e889bd81b4735ee5"}},{"code_sha256_prefix":"8c6a9f390c7af799","entry":"dict_apply_split","repo":"YanjieZe/Improved-3D-Diffusion-Policy","repo_kind":"listed","path":"Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/pytorch_util.py","file_url":"https://github.com/YanjieZe/Improved-3D-Diffusion-Policy/blob/HEAD/Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/pytorch_util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c6a9f390c7af799"}},{"code_sha256_prefix":"85ef80fba2caa71d","entry":"downsample_mask","repo":"YanjieZe/Improved-3D-Diffusion-Policy","repo_kind":"listed","path":"Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/sampler.py","file_url":"https://github.com/YanjieZe/Improved-3D-Diffusion-Policy/blob/HEAD/Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/sampler.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"85ef80fba2caa71d"}},{"code_sha256_prefix":"1d4ddad52e215d73","entry":"get_optimal_chunks","repo":"YanjieZe/Improved-3D-Diffusion-Policy","repo_kind":"listed","path":"Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/replay_buffer.py","file_url":"https://github.com/YanjieZe/Improved-3D-Diffusion-Policy/blob/HEAD/Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/replay_buffer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1d4ddad52e215d73"}},{"code_sha256_prefix":"ed689847c85c1288","entry":"get_val_mask","repo":"YanjieZe/Improved-3D-Diffusion-Policy","repo_kind":"listed","path":"Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/sampler.py","file_url":"https://github.com/YanjieZe/Improved-3D-Diffusion-Policy/blob/HEAD/Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/sampler.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed689847c85c1288"}},{"code_sha256_prefix":"ad05722d0bd12d1e","entry":"pad_remaining_dims","repo":"YanjieZe/Improved-3D-Diffusion-Policy","repo_kind":"listed","path":"Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/pytorch_util.py","file_url":"https://github.com/YanjieZe/Improved-3D-Diffusion-Policy/blob/HEAD/Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/pytorch_util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad05722d0bd12d1e"}},{"code_sha256_prefix":"8804b3947f63df99","entry":"eulerToQuat","repo":"YanjieZe/Improved-3D-Diffusion-Policy","repo_kind":"listed","path":"Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/rotation_util.py","file_url":"https://github.com/YanjieZe/Improved-3D-Diffusion-Policy/blob/HEAD/Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/rotation_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8804b3947f63df99"}},{"code_sha256_prefix":"33f3fbcce84b1aea","entry":"extract_eef_action","repo":"YanjieZe/Improved-3D-Diffusion-Policy","repo_kind":"listed","path":"Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/gr1_action_util.py","file_url":"https://github.com/YanjieZe/Improved-3D-Diffusion-Policy/blob/HEAD/Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/gr1_action_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"33f3fbcce84b1aea"}},{"code_sha256_prefix":"9a3a68bbfdb7296a","entry":"grid_sample_pcd","repo":"YanjieZe/Improved-3D-Diffusion-Policy","repo_kind":"listed","path":"Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/multi_realsense.py","file_url":"https://github.com/YanjieZe/Improved-3D-Diffusion-Policy/blob/HEAD/Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/multi_realsense.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a3a68bbfdb7296a"}},{"code_sha256_prefix":"2d2f983f25ce771d","entry":"joint25_to_joint32","repo":"YanjieZe/Improved-3D-Diffusion-Policy","repo_kind":"listed","path":"Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/gr1_action_util.py","file_url":"https://github.com/YanjieZe/Improved-3D-Diffusion-Policy/blob/HEAD/Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/gr1_action_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2d2f983f25ce771d"}},{"code_sha256_prefix":"55c36da9a4287996","entry":"joint32_to_joint25","repo":"YanjieZe/Improved-3D-Diffusion-Policy","repo_kind":"listed","path":"Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/gr1_action_util.py","file_url":"https://github.com/YanjieZe/Improved-3D-Diffusion-Policy/blob/HEAD/Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/gr1_action_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"55c36da9a4287996"}},{"code_sha256_prefix":"d0f1a2766743b387","entry":"quatToEuler","repo":"YanjieZe/Improved-3D-Diffusion-Policy","repo_kind":"listed","path":"Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/rotation_util.py","file_url":"https://github.com/YanjieZe/Improved-3D-Diffusion-Policy/blob/HEAD/Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/rotation_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d0f1a2766743b387"}},{"code_sha256_prefix":"d83f4930e9aa9202","entry":"read_json_log","repo":"YanjieZe/Improved-3D-Diffusion-Policy","repo_kind":"listed","path":"Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/json_logger.py","file_url":"https://github.com/YanjieZe/Improved-3D-Diffusion-Policy/blob/HEAD/Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/json_logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d83f4930e9aa9202"}},{"code_sha256_prefix":"7fe0ec7f00563412","entry":"yamlToDict","repo":"YanjieZe/Improved-3D-Diffusion-Policy","repo_kind":"listed","path":"Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/rotation_util.py","file_url":"https://github.com/YanjieZe/Improved-3D-Diffusion-Policy/blob/HEAD/Improved-3D-Diffusion-Policy/diffusion_policy_3d/common/rotation_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7fe0ec7f00563412"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}