{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/conceptfusion-open-set-multimodal-3d-mapping","title":"ConceptFusion: Open-set Multimodal 3D Mapping","arxiv_id":"2302.07241","date":"2023-02-14","proceeding":null,"authors":["Krishna Murthy Jatavallabhula","Alihusein Kuwajerwala","Qiao Gu","Mohd Omama","Tao Chen","Alaa Maalouf","Shuang Li","Ganesh Iyer","Soroush Saryazdi","Nikhil Keetha","Ayush Tewari","Joshua B. Tenenbaum","Celso Miguel de Melo","Madhava Krishna","Liam Paull","Florian Shkurti","Antonio Torralba"],"abstract":"Building 3D maps of the environment is central to robot navigation, planning, and interaction with objects in a scene. Most existing approaches that integrate semantic concepts with 3D maps largely remain confined to the closed-set setting: they can only reason about a finite set of concepts, pre-defined at training time. Further, these maps can only be queried using class labels, or in recent work, using text prompts. We address both these issues with ConceptFusion, a scene representation that is (1) fundamentally open-set, enabling reasoning beyond a closed set of concepts and (ii) inherently multimodal, enabling a diverse range of possible queries to the 3D map, from language, to images, to audio, to 3D geometry, all working in concert. ConceptFusion leverages the open-set capabilities of today's foundation models pre-trained on internet-scale data to reason about concepts across modalities such as natural language, images, and audio. We demonstrate that pixel-aligned open-set features can be fused into 3D maps via traditional SLAM and multi-view fusion approaches. This enables effective zero-shot spatial reasoning, not needing any additional training or finetuning, and retains long-tailed concepts better than supervised approaches, outperforming them by more than 40% margin on 3D IoU. We extensively evaluate ConceptFusion on a number of real-world datasets, simulated home environments, a real-world tabletop manipulation task, and an autonomous driving platform. We showcase new avenues for blending foundation models with 3D open-set multimodal mapping. For more information, visit our project page https://concept-fusion.github.io or watch our 5-minute explainer video https://www.youtube.com/watch?v=rkXgws8fiDs","url_abs":"https://arxiv.org/abs/2302.07241v3","url_pdf":"https://arxiv.org/pdf/2302.07241v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"conceptfusion-open-set-multimodal-3d-mapping","repo_url":"https://github.com/concept-fusion/concept-fusion","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"3d-geometry","task_name":"3D geometry"},{"task_slug":"autonomous-driving","task_name":"Autonomous Driving"},{"task_slug":"robot-navigation","task_name":"Robot Navigation"},{"task_slug":"spatial-reasoning","task_name":"Spatial Reasoning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2302.07241","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}