{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/3d-diffuser-actor-policy-diffusion-with-3d-1","title":"3D Diffuser Actor: Policy Diffusion with 3D Scene Representations","arxiv_id":"2402.10885","date":"2024-02-16","proceeding":null,"authors":["Tsung-Wei Ke","Nikolaos Gkanatsios","Katerina Fragkiadaki"],"abstract":"Diffusion policies are conditional diffusion models that learn robot action distributions conditioned on the robot and environment state. They have recently shown to outperform both deterministic and alternative action distribution learning formulations. 3D robot policies use 3D scene feature representations aggregated from a single or multiple camera views using sensed depth. They have shown to generalize better than their 2D counterparts across camera viewpoints. We unify these two lines of work and present 3D Diffuser Actor, a neural policy equipped with a novel 3D denoising transformer that fuses information from the 3D visual scene, a language instruction and proprioception to predict the noise in noised 3D robot pose trajectories. 3D Diffuser Actor sets a new state-of-the-art on RLBench with an absolute performance gain of 18.1% over the current SOTA on a multi-view setup and an absolute gain of 13.1% on a single-view setup. On the CALVIN benchmark, it improves over the current SOTA by a 9% relative increase. It also learns to control a robot manipulator in the real world from a handful of demonstrations. Through thorough comparisons with the current SOTA policies and ablations of our model, we show 3D Diffuser Actor's design choices dramatically outperform 2D representations, regression and classification objectives, absolute attentions, and holistic non-tokenized 3D scene embeddings.","url_abs":"https://arxiv.org/abs/2402.10885v3","url_pdf":"https://arxiv.org/pdf/2402.10885v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"3d-diffuser-actor-policy-diffusion-with-3d-1","repo_url":"https://github.com/nickgkan/3d_diffuser_actor","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"robot-manipulation","task_name":"Robot Manipulation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.10885","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.10885"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nickgkan/3d_diffuser_actor","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":9},"by_repo_kind":{"listed":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"82c73aecaae88536","entry":"all_gather","repo":"nickgkan/3d_diffuser_actor","repo_kind":"listed","path":"engine.py","file_url":"https://github.com/nickgkan/3d_diffuser_actor/blob/HEAD/engine.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"82c73aecaae88536"}},{"code_sha256_prefix":"caeb119cf55a786d","entry":"convert_efficient_fpn","repo":"nickgkan/3d_diffuser_actor","repo_kind":"listed","path":"diffuser_actor/utils/converter.py","file_url":"https://github.com/nickgkan/3d_diffuser_actor/blob/HEAD/diffuser_actor/utils/converter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"caeb119cf55a786d"}},{"code_sha256_prefix":"c921e2f2f2ff9812","entry":"convert_multihead_flash_attention","repo":"nickgkan/3d_diffuser_actor","repo_kind":"listed","path":"diffuser_actor/utils/converter.py","file_url":"https://github.com/nickgkan/3d_diffuser_actor/blob/HEAD/diffuser_actor/utils/converter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c921e2f2f2ff9812"}},{"code_sha256_prefix":"5122b84a1c2fb599","entry":"load_resnet18","repo":"nickgkan/3d_diffuser_actor","repo_kind":"listed","path":"diffuser_actor/utils/resnet.py","file_url":"https://github.com/nickgkan/3d_diffuser_actor/blob/HEAD/diffuser_actor/utils/resnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5122b84a1c2fb599"}},{"code_sha256_prefix":"592a9914f3d643b4","entry":"load_resnet50","repo":"nickgkan/3d_diffuser_actor","repo_kind":"listed","path":"diffuser_actor/utils/resnet.py","file_url":"https://github.com/nickgkan/3d_diffuser_actor/blob/HEAD/diffuser_actor/utils/resnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"592a9914f3d643b4"}},{"code_sha256_prefix":"616805ec8f4f95fb","entry":"multi_head_attention_forward","repo":"nickgkan/3d_diffuser_actor","repo_kind":"listed","path":"diffuser_actor/utils/multihead_custom_attention.py","file_url":"https://github.com/nickgkan/3d_diffuser_actor/blob/HEAD/diffuser_actor/utils/multihead_custom_attention.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"616805ec8f4f95fb"}},{"code_sha256_prefix":"bad211cd427d59bf","entry":"norm_tensor","repo":"nickgkan/3d_diffuser_actor","repo_kind":"listed","path":"diffuser_actor/utils/utils.py","file_url":"https://github.com/nickgkan/3d_diffuser_actor/blob/HEAD/diffuser_actor/utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bad211cd427d59bf"}},{"code_sha256_prefix":"6a887764a925a2e4","entry":"normalise_quat","repo":"nickgkan/3d_diffuser_actor","repo_kind":"listed","path":"diffuser_actor/utils/utils.py","file_url":"https://github.com/nickgkan/3d_diffuser_actor/blob/HEAD/diffuser_actor/utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6a887764a925a2e4"}},{"code_sha256_prefix":"5ee893d8fc620de5","entry":"sample_ghost_points_grid","repo":"nickgkan/3d_diffuser_actor","repo_kind":"listed","path":"diffuser_actor/utils/utils.py","file_url":"https://github.com/nickgkan/3d_diffuser_actor/blob/HEAD/diffuser_actor/utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5ee893d8fc620de5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}