{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/clio-real-time-task-driven-open-set-3d-scene","title":"Clio: Real-time Task-Driven Open-Set 3D Scene Graphs","arxiv_id":"2404.13696","date":"2024-04-21","proceeding":null,"authors":["Dominic Maggio","Yun Chang","Nathan Hughes","Matthew Trang","Dan Griffith","Carlyn Dougherty","Eric Cristofalo","Lukas Schmid","Luca Carlone"],"abstract":"Modern tools for class-agnostic image segmentation (e.g., SegmentAnything) and open-set semantic understanding (e.g., CLIP) provide unprecedented opportunities for robot perception and mapping. While traditional closed-set metric-semantic maps were restricted to tens or hundreds of semantic classes, we can now build maps with a plethora of objects and countless semantic variations. This leaves us with a fundamental question: what is the right granularity for the objects (and, more generally, for the semantic concepts) the robot has to include in its map representation? While related work implicitly chooses a level of granularity by tuning thresholds for object detection, we argue that such a choice is intrinsically task-dependent. The first contribution of this paper is to propose a task-driven 3D scene understanding problem, where the robot is given a list of tasks in natural language and has to select the granularity and the subset of objects and scene structure to retain in its map that is sufficient to complete the tasks. We show that this problem can be naturally formulated using the Information Bottleneck (IB), an established information-theoretic framework. The second contribution is an algorithm for task-driven 3D scene understanding based on an Agglomerative IB approach, that is able to cluster 3D primitives in the environment into task-relevant objects and regions and executes incrementally. The third contribution is to integrate our task-driven clustering algorithm into a real-time pipeline, named Clio, that constructs a hierarchical 3D scene graph of the environment online using only onboard compute, as the robot explores it. Our final contribution is an extensive experimental campaign showing that Clio not only allows real-time construction of compact open-set 3D scene graphs, but also improves the accuracy of task execution by limiting the map to relevant semantic concepts.","url_abs":"https://arxiv.org/abs/2404.13696v4","url_pdf":"https://arxiv.org/pdf/2404.13696v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"clio-real-time-task-driven-open-set-3d-scene","repo_url":"https://github.com/mit-spark/clio","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"BSD-2-Clause"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.13696","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.13696"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mit-spark/clio","reach":{"status":"ok","spdx":"BSD-2-Clause"}}],"summary":{"ran":15,"unverified":2},"by_repo_kind":{"official":{"samples":17,"ran":15,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0f04babc506ca3a1","entry":"compute_cosine_sim","repo":"mit-spark/clio","repo_kind":"official","path":"clio_batch/helpers.py","file_url":"https://github.com/mit-spark/clio/blob/HEAD/clio_batch/helpers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"0f04babc506ca3a1"}},{"code_sha256_prefix":"904d92e80020dd5b","entry":"compute_sim_to_tasks","repo":"mit-spark/clio","repo_kind":"official","path":"clio_batch/helpers.py","file_url":"https://github.com/mit-spark/clio/blob/HEAD/clio_batch/helpers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"904d92e80020dd5b"}},{"code_sha256_prefix":"c20ac8b4610fedb0","entry":"contains_object","repo":"mit-spark/clio","repo_kind":"official","path":"clio_batch/object_cluster.py","file_url":"https://github.com/mit-spark/clio/blob/HEAD/clio_batch/object_cluster.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"c20ac8b4610fedb0"}},{"code_sha256_prefix":"cace94fae8881fde","entry":"create_cluster_graphs","repo":"mit-spark/clio","repo_kind":"official","path":"clio_batch/object_cluster.py","file_url":"https://github.com/mit-spark/clio/blob/HEAD/clio_batch/object_cluster.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"cace94fae8881fde"}},{"code_sha256_prefix":"0afd6939ef1f5ea5","entry":"create_obb_from_dict","repo":"mit-spark/clio","repo_kind":"official","path":"clio_data_maker/rescale_bbox.py","file_url":"https://github.com/mit-spark/clio/blob/HEAD/clio_data_maker/rescale_bbox.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"0afd6939ef1f5ea5"}},{"code_sha256_prefix":"81cc50e1fa115ce7","entry":"default_device","repo":"mit-spark/clio","repo_kind":"official","path":"clio_batch/helpers.py","file_url":"https://github.com/mit-spark/clio/blob/HEAD/clio_batch/helpers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"81cc50e1fa115ce7"}},{"code_sha256_prefix":"b18d5a985faa6004","entry":"get_box_verticies","repo":"mit-spark/clio","repo_kind":"official","path":"clio_eval/iou.py","file_url":"https://github.com/mit-spark/clio/blob/HEAD/clio_eval/iou.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"b18d5a985faa6004"}},{"code_sha256_prefix":"89c8f9177bf25da3","entry":"get_dsg_version","repo":"mit-spark/clio","repo_kind":"official","path":"clio_eval/evaluate_helpers.py","file_url":"https://github.com/mit-spark/clio/blob/HEAD/clio_eval/evaluate_helpers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"89c8f9177bf25da3"}},{"code_sha256_prefix":"308809ffeae78b00","entry":"get_last_feature","repo":"mit-spark/clio","repo_kind":"official","path":"clio_batch/cluster_utilities.py","file_url":"https://github.com/mit-spark/clio/blob/HEAD/clio_batch/cluster_utilities.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"308809ffeae78b00"}},{"code_sha256_prefix":"85f1beae457883d3","entry":"is_point_inside_box","repo":"mit-spark/clio","repo_kind":"official","path":"clio_eval/utils.py","file_url":"https://github.com/mit-spark/clio/blob/HEAD/clio_eval/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"85f1beae457883d3"}},{"code_sha256_prefix":"2292882fb71b75af","entry":"max_from_box_position","repo":"mit-spark/clio","repo_kind":"official","path":"clio_eval/utils.py","file_url":"https://github.com/mit-spark/clio/blob/HEAD/clio_eval/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"2292882fb71b75af"}},{"code_sha256_prefix":"1c3914ffaf0ba4d0","entry":"min_from_box_position","repo":"mit-spark/clio","repo_kind":"official","path":"clio_eval/utils.py","file_url":"https://github.com/mit-spark/clio/blob/HEAD/clio_eval/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"1c3914ffaf0ba4d0"}},{"code_sha256_prefix":"109cd6e7b1ebd161","entry":"mutual_information","repo":"mit-spark/clio","repo_kind":"official","path":"clio_batch/information_metrics.py","file_url":"https://github.com/mit-spark/clio/blob/HEAD/clio_batch/information_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"109cd6e7b1ebd161"}},{"code_sha256_prefix":"1f9d9ed8032cec88","entry":"rescale","repo":"mit-spark/clio","repo_kind":"official","path":"clio_data_maker/rescale_bbox.py","file_url":"https://github.com/mit-spark/clio/blob/HEAD/clio_data_maker/rescale_bbox.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"1f9d9ed8032cec88"}},{"code_sha256_prefix":"31fec44a12e1775a","entry":"shannon_entropy","repo":"mit-spark/clio","repo_kind":"official","path":"clio_batch/information_metrics.py","file_url":"https://github.com/mit-spark/clio/blob/HEAD/clio_batch/information_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"31fec44a12e1775a"}},{"code_sha256_prefix":"958a2e2335058ed4","entry":"bboxes_intersect","repo":"mit-spark/clio","repo_kind":"official","path":"clio_batch/object_cluster.py","file_url":"https://github.com/mit-spark/clio/blob/HEAD/clio_batch/object_cluster.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"958a2e2335058ed4"}},{"code_sha256_prefix":"81689c2e5f8bb283","entry":"js_divergence","repo":"mit-spark/clio","repo_kind":"official","path":"clio_batch/information_metrics.py","file_url":"https://github.com/mit-spark/clio/blob/HEAD/clio_batch/information_metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"81689c2e5f8bb283"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}