{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/point-cloud-matters-rethinking-the-impact-of","title":"Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learning","arxiv_id":"2402.02500","date":"2024-02-04","proceeding":null,"authors":["Haoyi Zhu","Yating Wang","Di Huang","Weicai Ye","Wanli Ouyang","Tong He"],"abstract":"In robot learning, the observation space is crucial due to the distinct characteristics of different modalities, which can potentially become a bottleneck alongside policy design. In this study, we explore the influence of various observation spaces on robot learning, focusing on three predominant modalities: RGB, RGB-D, and point cloud. We introduce OBSBench, a benchmark comprising two simulators and 125 tasks, along with standardized pipelines for various encoders and policy baselines. Extensive experiments on diverse contact-rich manipulation tasks reveal a notable trend: point cloud-based methods, even those with the simplest designs, frequently outperform their RGB and RGB-D counterparts. This trend persists in both scenarios: training from scratch and utilizing pre-training. Furthermore, our findings demonstrate that point cloud observations often yield better policy performance and significantly stronger generalization capabilities across various geometric and visual conditions. These outcomes suggest that the 3D point cloud is a valuable observation modality for intricate robotic tasks. We also suggest that incorporating both appearance and coordinate information can enhance the performance of point cloud methods. We hope our work provides valuable insights and guidance for designing more generalizable and robust robotic models. Codes are available at https://github.com/HaoyiZhu/PointCloudMatters.","url_abs":"https://arxiv.org/abs/2402.02500v3","url_pdf":"https://arxiv.org/pdf/2402.02500v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"point-cloud-matters-rethinking-the-impact-of","repo_url":"https://github.com/haoyizhu/pointcloudmatters","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"point-cloud-matters-rethinking-the-impact-of","repo_url":"https://github.com/haoyizhu/realrobot","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"contact-rich-manipulation","task_name":"Contact-rich Manipulation"},{"task_slug":"zero-shot-generalization","task_name":"Zero-shot Generalization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.02500","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.02500"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/HaoyiZhu/PointCloudMatters","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/haoyizhu/pointcloudmatters","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/haoyizhu/realrobot","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":8,"unverified":1},"by_repo_kind":{"official":{"samples":9,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a488db5b59ecea96","entry":"check_state_dict","repo":"HaoyiZhu/PointCloudMatters","repo_kind":"official","path":"src/utils/common_utils.py","file_url":"https://github.com/HaoyiZhu/PointCloudMatters/blob/HEAD/src/utils/common_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a488db5b59ecea96"}},{"code_sha256_prefix":"0bccf0e06e6688aa","entry":"import_modules_from_strings","repo":"HaoyiZhu/PointCloudMatters","repo_kind":"official","path":"src/utils/misc.py","file_url":"https://github.com/HaoyiZhu/PointCloudMatters/blob/HEAD/src/utils/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0bccf0e06e6688aa"}},{"code_sha256_prefix":"129f75ac28d5d99f","entry":"interpolate_linear","repo":"HaoyiZhu/PointCloudMatters","repo_kind":"official","path":"src/utils/misc.py","file_url":"https://github.com/HaoyiZhu/PointCloudMatters/blob/HEAD/src/utils/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"129f75ac28d5d99f"}},{"code_sha256_prefix":"3551b62f3276001a","entry":"is_seq_of","repo":"HaoyiZhu/PointCloudMatters","repo_kind":"official","path":"src/utils/misc.py","file_url":"https://github.com/HaoyiZhu/PointCloudMatters/blob/HEAD/src/utils/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3551b62f3276001a"}},{"code_sha256_prefix":"d9118aea6a0811f4","entry":"load_bytes","repo":"HaoyiZhu/PointCloudMatters","repo_kind":"official","path":"src/utils/io.py","file_url":"https://github.com/HaoyiZhu/PointCloudMatters/blob/HEAD/src/utils/io.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d9118aea6a0811f4"}},{"code_sha256_prefix":"7843f915d988dd5e","entry":"load_numpy_text","repo":"HaoyiZhu/PointCloudMatters","repo_kind":"official","path":"src/utils/io.py","file_url":"https://github.com/HaoyiZhu/PointCloudMatters/blob/HEAD/src/utils/io.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7843f915d988dd5e"}},{"code_sha256_prefix":"76e8fea908592e80","entry":"load_text","repo":"HaoyiZhu/PointCloudMatters","repo_kind":"official","path":"src/utils/io.py","file_url":"https://github.com/HaoyiZhu/PointCloudMatters/blob/HEAD/src/utils/io.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"76e8fea908592e80"}},{"code_sha256_prefix":"0d86269cf8e78d35","entry":"reduce_mean","repo":"HaoyiZhu/PointCloudMatters","repo_kind":"official","path":"src/utils/common_utils.py","file_url":"https://github.com/HaoyiZhu/PointCloudMatters/blob/HEAD/src/utils/common_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d86269cf8e78d35"}},{"code_sha256_prefix":"f7898a5762a4d3e4","entry":"multi_apply","repo":"HaoyiZhu/PointCloudMatters","repo_kind":"official","path":"src/utils/common_utils.py","file_url":"https://github.com/HaoyiZhu/PointCloudMatters/blob/HEAD/src/utils/common_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f7898a5762a4d3e4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}