{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/visual-reinforcement-learning-with-self","title":"Visual Reinforcement Learning with Self-Supervised 3D Representations","arxiv_id":"2210.07241","date":"2022-10-13","proceeding":null,"authors":["Yanjie Ze","Nicklas Hansen","Yinbo Chen","Mohit Jain","Xiaolong Wang"],"abstract":"A prominent approach to visual Reinforcement Learning (RL) is to learn an internal state representation using self-supervised methods, which has the potential benefit of improved sample-efficiency and generalization through additional learning signal and inductive biases. However, while the real world is inherently 3D, prior efforts have largely been focused on leveraging 2D computer vision techniques as auxiliary self-supervision. In this work, we present a unified framework for self-supervised learning of 3D representations for motor control. Our proposed framework consists of two phases: a pretraining phase where a deep voxel-based 3D autoencoder is pretrained on a large object-centric dataset, and a finetuning phase where the representation is jointly finetuned together with RL on in-domain data. We empirically show that our method enjoys improved sample efficiency in simulated manipulation tasks compared to 2D representation learning methods. Additionally, our learned policies transfer zero-shot to a real robot setup with only approximate geometric correspondence, and successfully solve motor control tasks that involve grasping and lifting from a single, uncalibrated RGB camera. Code and videos are available at https://yanjieze.com/3d4rl/ .","url_abs":"https://arxiv.org/abs/2210.07241v2","url_pdf":"https://arxiv.org/pdf/2210.07241v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"visual-reinforcement-learning-with-self","repo_url":"https://github.com/YanjieZe/rl3d","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.07241","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.07241"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/YanjieZe/rl3d","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"ran_draft_wrong":1,"ran_fixture":1,"unverified":11},"by_repo_kind":{"official":{"samples":17,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6c2c84b4ccb7f17b","entry":"cat","repo":"YanjieZe/rl3d","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/YanjieZe/rl3d/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6c2c84b4ccb7f17b"}},{"code_sha256_prefix":"2df17488b927be0b","entry":"gaussian_logprob","repo":"YanjieZe/rl3d","repo_kind":"official","path":"src/algorithms/modules.py","file_url":"https://github.com/YanjieZe/rl3d/blob/HEAD/src/algorithms/modules.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2df17488b927be0b"}},{"code_sha256_prefix":"718e1902f65f1be2","entry":"hsv2rgb","repo":"YanjieZe/rl3d","repo_kind":"official","path":"src/color_jitter.py","file_url":"https://github.com/YanjieZe/rl3d/blob/HEAD/src/color_jitter.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"718e1902f65f1be2"}},{"code_sha256_prefix":"1ac4e5bffc3dbbd3","entry":"make_dir","repo":"YanjieZe/rl3d","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/YanjieZe/rl3d/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1ac4e5bffc3dbbd3"}},{"code_sha256_prefix":"1c24bb2fb72c4f9d","entry":"rgb2hsv","repo":"YanjieZe/rl3d","repo_kind":"official","path":"src/color_jitter.py","file_url":"https://github.com/YanjieZe/rl3d/blob/HEAD/src/color_jitter.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1c24bb2fb72c4f9d"}},{"code_sha256_prefix":"121cb336a1e1278a","entry":"squash","repo":"YanjieZe/rl3d","repo_kind":"official","path":"src/algorithms/modules.py","file_url":"https://github.com/YanjieZe/rl3d/blob/HEAD/src/algorithms/modules.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"121cb336a1e1278a"}},{"code_sha256_prefix":"64155049b60f0d6b","entry":"conv","repo":"YanjieZe/rl3d","repo_kind":"official","path":"src/algorithms/modules_3d.py","file_url":"https://github.com/YanjieZe/rl3d/blob/HEAD/src/algorithms/modules_3d.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"64155049b60f0d6b"}},{"code_sha256_prefix":"797689fffbce98da","entry":"euler2mat","repo":"YanjieZe/rl3d","repo_kind":"official","path":"src/algorithms/rot_utils.py","file_url":"https://github.com/YanjieZe/rl3d/blob/HEAD/src/algorithms/rot_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"797689fffbce98da"}},{"code_sha256_prefix":"5912a589c21e0e7f","entry":"load_config","repo":"YanjieZe/rl3d","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/YanjieZe/rl3d/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5912a589c21e0e7f"}},{"code_sha256_prefix":"a935467503363d78","entry":"prefill","repo":"YanjieZe/rl3d","repo_kind":"official","path":"src/algorithms/per.py","file_url":"https://github.com/YanjieZe/rl3d/blob/HEAD/src/algorithms/per.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a935467503363d78"}},{"code_sha256_prefix":"8cb3ce3c5bb986f0","entry":"prefill_memory","repo":"YanjieZe/rl3d","repo_kind":"official","path":"src/algorithms/per.py","file_url":"https://github.com/YanjieZe/rl3d/blob/HEAD/src/algorithms/per.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8cb3ce3c5bb986f0"}},{"code_sha256_prefix":"a0a7fcf830f831b6","entry":"quat2euler","repo":"YanjieZe/rl3d","repo_kind":"official","path":"src/algorithms/rot_utils.py","file_url":"https://github.com/YanjieZe/rl3d/blob/HEAD/src/algorithms/rot_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a0a7fcf830f831b6"}},{"code_sha256_prefix":"86eaf46a4e1b7005","entry":"quat2mat","repo":"YanjieZe/rl3d","repo_kind":"official","path":"src/algorithms/rot_utils.py","file_url":"https://github.com/YanjieZe/rl3d/blob/HEAD/src/algorithms/rot_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"86eaf46a4e1b7005"}},{"code_sha256_prefix":"0604211861b8f1e9","entry":"random_conv","repo":"YanjieZe/rl3d","repo_kind":"official","path":"src/augmentations.py","file_url":"https://github.com/YanjieZe/rl3d/blob/HEAD/src/augmentations.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0604211861b8f1e9"}},{"code_sha256_prefix":"8ec40d7c6f748f4e","entry":"stn","repo":"YanjieZe/rl3d","repo_kind":"official","path":"src/algorithms/modules_3d.py","file_url":"https://github.com/YanjieZe/rl3d/blob/HEAD/src/algorithms/modules_3d.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8ec40d7c6f748f4e"}},{"code_sha256_prefix":"05f8b4749415d195","entry":"visual_representation","repo":"YanjieZe/rl3d","repo_kind":"official","path":"load_3d.py","file_url":"https://github.com/YanjieZe/rl3d/blob/HEAD/load_3d.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"05f8b4749415d195"}},{"code_sha256_prefix":"c71774582015fd4b","entry":"xavier_uniform_init","repo":"YanjieZe/rl3d","repo_kind":"official","path":"src/algorithms/modules.py","file_url":"https://github.com/YanjieZe/rl3d/blob/HEAD/src/algorithms/modules.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c71774582015fd4b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}