{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unsupervised-learning-of-visual-3d-keypoints","title":"Unsupervised Learning of Visual 3D Keypoints for Control","arxiv_id":"2106.07643","date":"2021-06-14","proceeding":null,"authors":["Boyuan Chen","Pieter Abbeel","Deepak Pathak"],"abstract":"Learning sensorimotor control policies from high-dimensional images crucially relies on the quality of the underlying visual representations. Prior works show that structured latent space such as visual keypoints often outperforms unstructured representations for robotic control. However, most of these representations, whether structured or unstructured are learned in a 2D space even though the control tasks are usually performed in a 3D environment. In this work, we propose a framework to learn such a 3D geometric structure directly from images in an end-to-end unsupervised manner. The input images are embedded into latent 3D keypoints via a differentiable encoder which is trained to optimize both a multi-view consistency loss and downstream task objective. These discovered 3D keypoints tend to meaningfully capture robot joints as well as object movements in a consistent manner across both time and 3D space. The proposed approach outperforms prior state-of-art methods across a variety of reinforcement learning benchmarks. Code and videos at https://buoyancy99.github.io/unsup-3d-keypoints/","url_abs":"https://arxiv.org/abs/2106.07643v1","url_pdf":"https://arxiv.org/pdf/2106.07643v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unsupervised-learning-of-visual-3d-keypoints","repo_url":"https://github.com/buoyancy99/unsup-3d-keypoints","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.07643","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.07643"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/buoyancy99/unsup-3d-keypoints","reach":null}],"summary":{"ran":2,"unverified":4},"by_repo_kind":{"official":{"samples":6,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f2205d997c25f553","entry":"CustomeDecoder","repo":"buoyancy99/unsup-3d-keypoints","repo_kind":"official","path":"algorithms/common/models/keypoint_net.py","file_url":"https://github.com/buoyancy99/unsup-3d-keypoints/blob/HEAD/algorithms/common/models/keypoint_net.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2205d997c25f553"}},{"code_sha256_prefix":"5546c8972dde0b92","entry":"CustomeEncoder","repo":"buoyancy99/unsup-3d-keypoints","repo_kind":"official","path":"algorithms/common/models/keypoint_net.py","file_url":"https://github.com/buoyancy99/unsup-3d-keypoints/blob/HEAD/algorithms/common/models/keypoint_net.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5546c8972dde0b92"}},{"code_sha256_prefix":"b9c4a101ff5bb3af","entry":"BaseCNN","repo":"buoyancy99/unsup-3d-keypoints","repo_kind":"official","path":"algorithms/common/models/keypoint_net.py","file_url":"https://github.com/buoyancy99/unsup-3d-keypoints/blob/HEAD/algorithms/common/models/keypoint_net.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b9c4a101ff5bb3af"}},{"code_sha256_prefix":"814a68a0d1a4f32a","entry":"KeypointNet3d","repo":"buoyancy99/unsup-3d-keypoints","repo_kind":"official","path":"algorithms/common/models/keypoint_net.py","file_url":"https://github.com/buoyancy99/unsup-3d-keypoints/blob/HEAD/algorithms/common/models/keypoint_net.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"814a68a0d1a4f32a"}},{"code_sha256_prefix":"0d36d42fc4bc30fb","entry":"KeypointNetBase","repo":"buoyancy99/unsup-3d-keypoints","repo_kind":"official","path":"algorithms/common/models/keypoint_net.py","file_url":"https://github.com/buoyancy99/unsup-3d-keypoints/blob/HEAD/algorithms/common/models/keypoint_net.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d36d42fc4bc30fb"}},{"code_sha256_prefix":"174c85c073dd49b0","entry":"get_cmap","repo":"buoyancy99/unsup-3d-keypoints","repo_kind":"official","path":"algorithms/common/models/keypoint_net.py","file_url":"https://github.com/buoyancy99/unsup-3d-keypoints/blob/HEAD/algorithms/common/models/keypoint_net.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"174c85c073dd49b0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}