{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/3d-shape-reconstruction-from-vision-and-touch","title":"3D Shape Reconstruction from Vision and Touch","arxiv_id":"2007.03778","date":"2020-07-07","proceeding":"NeurIPS 2020 12","authors":["Edward J. Smith","Roberto Calandra","Adriana Romero","Georgia Gkioxari","David Meger","Jitendra Malik","Michal Drozdzal"],"abstract":"When a toddler is presented a new toy, their instinctual behaviour is to pick it upand inspect it with their hand and eyes in tandem, clearly searching over its surface to properly understand what they are playing with. At any instance here, touch provides high fidelity localized information while vision provides complementary global context. However, in 3D shape reconstruction, the complementary fusion of visual and haptic modalities remains largely unexplored. In this paper, we study this problem and present an effective chart-based approach to multi-modal shape understanding which encourages a similar fusion vision and touch information.To do so, we introduce a dataset of simulated touch and vision signals from the interaction between a robotic hand and a large array of 3D objects. Our results show that (1) leveraging both vision and touch signals consistently improves single-modality baselines; (2) our approach outperforms alternative modality fusion methods and strongly benefits from the proposed chart-based structure; (3) there construction quality increases with the number of grasps provided; and (4) the touch information not only enhances the reconstruction at the touch site but also extrapolates to its local neighborhood.","url_abs":"https://arxiv.org/abs/2007.03778v2","url_pdf":"https://arxiv.org/pdf/2007.03778v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"3d-shape-reconstruction-from-vision-and-touch","repo_url":"https://github.com/facebookresearch/3D-Vision-and-Touch","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"3d-shape-reconstruction","task_name":"3D Shape Reconstruction"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2007.03778","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2007.03778"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/3D-Vision-and-Touch","reach":null}],"summary":{"ran_draft_wrong":1,"ran":3,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"519f40bf3f792dbb","entry":"CNN_layer","repo":"facebookresearch/3D-Vision-and-Touch","repo_kind":"official","path":"vision_charts/models.py","file_url":"https://github.com/facebookresearch/3D-Vision-and-Touch/blob/HEAD/vision_charts/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"519f40bf3f792dbb"}},{"code_sha256_prefix":"dadbe75f331a2b52","entry":"GCN","repo":"facebookresearch/3D-Vision-and-Touch","repo_kind":"official","path":"vision_charts/models.py","file_url":"https://github.com/facebookresearch/3D-Vision-and-Touch/blob/HEAD/vision_charts/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"dadbe75f331a2b52"}},{"code_sha256_prefix":"4ec071a89e42f63c","entry":"GCN_layer","repo":"facebookresearch/3D-Vision-and-Touch","repo_kind":"official","path":"vision_charts/models.py","file_url":"https://github.com/facebookresearch/3D-Vision-and-Touch/blob/HEAD/vision_charts/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4ec071a89e42f63c"}},{"code_sha256_prefix":"3f36c29acc4a981e","entry":"Image_Encoder","repo":"facebookresearch/3D-Vision-and-Touch","repo_kind":"official","path":"vision_charts/models.py","file_url":"https://github.com/facebookresearch/3D-Vision-and-Touch/blob/HEAD/vision_charts/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3f36c29acc4a981e"}},{"code_sha256_prefix":"b72dde989b5d485e","entry":"Encoder","repo":"facebookresearch/3D-Vision-and-Touch","repo_kind":"official","path":"vision_charts/models.py","file_url":"https://github.com/facebookresearch/3D-Vision-and-Touch/blob/HEAD/vision_charts/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b72dde989b5d485e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}