{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/full-body-visual-self-modeling-of-robot","title":"Full-Body Visual Self-Modeling of Robot Morphologies","arxiv_id":"2111.06389","date":"2021-11-11","proceeding":null,"authors":["Boyuan Chen","Robert Kwiatkowski","Carl Vondrick","Hod Lipson"],"abstract":"Internal computational models of physical bodies are fundamental to the ability of robots and animals alike to plan and control their actions. These \"self-models\" allow robots to consider outcomes of multiple possible future actions, without trying them out in physical reality. Recent progress in fully data-driven self-modeling has enabled machines to learn their own forward kinematics directly from task-agnostic interaction data. However, forward-kinema\\-tics models can only predict limited aspects of the morphology, such as the position of end effectors or velocity of joints and masses. A key challenge is to model the entire morphology and kinematics, without prior knowledge of what aspects of the morphology will be relevant to future tasks. Here, we propose that instead of directly modeling forward-kinematics, a more useful form of self-modeling is one that could answer space occupancy queries, conditioned on the robot's state. Such query-driven self models are continuous in the spatial domain, memory efficient, fully differentiable and kinematic aware. In physical experiments, we demonstrate how a visual self-model is accurate to about one percent of the workspace, enabling the robot to perform various motion planning and control tasks. Visual self-modeling can also allow the robot to detect, localize and recover from real-world damage, leading to improved machine resiliency. Our project website is at: https://robot-morphology.cs.columbia.edu/","url_abs":"https://arxiv.org/abs/2111.06389v2","url_pdf":"https://arxiv.org/pdf/2111.06389v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"full-body-visual-self-modeling-of-robot","repo_url":"https://github.com/BoyuanChen/visual-selfmodeling","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"motion-planning","task_name":"Motion Planning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2111.06389","atlas_url":"https://app.syntology.ai/?focus=2111.06389","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2111.06389"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/BoyuanChen/visual-selfmodeling","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"unverified":2},"by_repo_kind":{"official":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2a0c158381f5db93","entry":"compute_unit_sphere_transform","repo":"BoyuanChen/visual-selfmodeling","repo_kind":"official","path":"utils/common.py","file_url":"https://github.com/BoyuanChen/visual-selfmodeling/blob/HEAD/utils/common.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2a0c158381f5db93"}},{"code_sha256_prefix":"f3a2b8fb9bd02cc7","entry":"get_body_colors","repo":"BoyuanChen/visual-selfmodeling","repo_kind":"official","path":"utils/common.py","file_url":"https://github.com/BoyuanChen/visual-selfmodeling/blob/HEAD/utils/common.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f3a2b8fb9bd02cc7"}},{"code_sha256_prefix":"4c74437ab5ad2ac8","entry":"get_view_frustum","repo":"BoyuanChen/visual-selfmodeling","repo_kind":"official","path":"fusion.py","file_url":"https://github.com/BoyuanChen/visual-selfmodeling/blob/HEAD/fusion.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4c74437ab5ad2ac8"}},{"code_sha256_prefix":"cbbf6e45e2f6ff75","entry":"load_config","repo":"BoyuanChen/visual-selfmodeling","repo_kind":"official","path":"utils/eval_mesh.py","file_url":"https://github.com/BoyuanChen/visual-selfmodeling/blob/HEAD/utils/eval_mesh.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cbbf6e45e2f6ff75"}},{"code_sha256_prefix":"faf47cf1a0889190","entry":"rename_ckpt_for_multi_models","repo":"BoyuanChen/visual-selfmodeling","repo_kind":"official","path":"models.py","file_url":"https://github.com/BoyuanChen/visual-selfmodeling/blob/HEAD/models.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"faf47cf1a0889190"}},{"code_sha256_prefix":"a1150c1b03c31b42","entry":"rigid_transform","repo":"BoyuanChen/visual-selfmodeling","repo_kind":"official","path":"fusion.py","file_url":"https://github.com/BoyuanChen/visual-selfmodeling/blob/HEAD/fusion.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a1150c1b03c31b42"}},{"code_sha256_prefix":"3c5c35d8c716ebad","entry":"check_mesh_contains","repo":"BoyuanChen/visual-selfmodeling","repo_kind":"official","path":"utils/libmesh/inside_mesh.py","file_url":"https://github.com/BoyuanChen/visual-selfmodeling/blob/HEAD/utils/libmesh/inside_mesh.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c5c35d8c716ebad"}},{"code_sha256_prefix":"72e1c034735c23ca","entry":"gradient","repo":"BoyuanChen/visual-selfmodeling","repo_kind":"official","path":"utils/common.py","file_url":"https://github.com/BoyuanChen/visual-selfmodeling/blob/HEAD/utils/common.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"72e1c034735c23ca"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}