{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/open3dsg-open-vocabulary-3d-scene-graphs-from","title":"Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationships","arxiv_id":"2402.12259","date":"2024-02-19","proceeding":"CVPR 2024 1","authors":["Sebastian Koch","Narunas Vaskevicius","Mirco Colosi","Pedro Hermosilla","Timo Ropinski"],"abstract":"Current approaches for 3D scene graph prediction rely on labeled datasets to train models for a fixed set of known object classes and relationship categories. We present Open3DSG, an alternative approach to learn 3D scene graph prediction in an open world without requiring labeled scene graph data. We co-embed the features from a 3D scene graph prediction backbone with the feature space of powerful open world 2D vision language foundation models. This enables us to predict 3D scene graphs from 3D point clouds in a zero-shot manner by querying object classes from an open vocabulary and predicting the inter-object relationships from a grounded LLM with scene graph features and queried object classes as context. Open3DSG is the first 3D point cloud method to predict not only explicit open-vocabulary object classes, but also open-set relationships that are not limited to a predefined label set, making it possible to express rare as well as specific objects and relationships in the predicted 3D scene graph. Our experiments show that Open3DSG is effective at predicting arbitrary object classes as well as their complex inter-object relationships describing spatial, supportive, semantic and comparative relationships.","url_abs":"https://arxiv.org/abs/2402.12259v2","url_pdf":"https://arxiv.org/pdf/2402.12259v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"open3dsg-open-vocabulary-3d-scene-graphs-from","repo_url":"https://github.com/boschresearch/Open3DSG","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"AGPL-3.0"}}],"tasks":[{"task_slug":"3d-scene-graph-generation","task_name":"3d scene graph generation"},{"task_slug":"object","task_name":"Object"},{"task_slug":"prediction","task_name":"Prediction"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.12259","atlas_url":"https://app.syntology.ai/?focus=2402.12259","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.12259"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/boschresearch/Open3DSG","reach":{"status":"ok","spdx":"AGPL-3.0"}}],"summary":{"ran_draft_wrong":1,"ran":7,"ran_honours":1,"ran_fixture":1,"unverified":2},"by_repo_kind":{"official":{"samples":12,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"d5fa2a668d4411e5","entry":"build_mlp","repo":"boschresearch/Open3DSG","repo_kind":"official","path":"open3dsg/models/layers.py","file_url":"https://github.com/boschresearch/Open3DSG/blob/HEAD/open3dsg/models/layers.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"d5fa2a668d4411e5"}},{"code_sha256_prefix":"8bb0bcb84a7b548c","entry":"build_mlp","repo":"boschresearch/Open3DSG","repo_kind":"official","path":"open3dsg/models/network_util.py","file_url":"https://github.com/boschresearch/Open3DSG/blob/HEAD/open3dsg/models/network_util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"8bb0bcb84a7b548c"}},{"code_sha256_prefix":"e996c64ee89d2b50","entry":"feature_transform_regularizer","repo":"boschresearch/Open3DSG","repo_kind":"official","path":"open3dsg/models/network_PointNet.py","file_url":"https://github.com/boschresearch/Open3DSG/blob/HEAD/open3dsg/models/network_PointNet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"e996c64ee89d2b50"}},{"code_sha256_prefix":"5255beb6ffa2726c","entry":"filter_args","repo":"boschresearch/Open3DSG","repo_kind":"official","path":"open3dsg/models/network_GNN.py","file_url":"https://github.com/boschresearch/Open3DSG/blob/HEAD/open3dsg/models/network_GNN.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"5255beb6ffa2726c"}},{"code_sha256_prefix":"01a860b46eded23f","entry":"filter_args_create","repo":"boschresearch/Open3DSG","repo_kind":"official","path":"open3dsg/models/network_GNN.py","file_url":"https://github.com/boschresearch/Open3DSG/blob/HEAD/open3dsg/models/network_GNN.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"01a860b46eded23f"}},{"code_sha256_prefix":"65aaa3be90604371","entry":"gen_descriptor","repo":"boschresearch/Open3DSG","repo_kind":"official","path":"open3dsg/models/op_utils.py","file_url":"https://github.com/boschresearch/Open3DSG/blob/HEAD/open3dsg/models/op_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"65aaa3be90604371"}},{"code_sha256_prefix":"ec413739d406e611","entry":"pc_normalize","repo":"boschresearch/Open3DSG","repo_kind":"official","path":"open3dsg/models/pointnet2_utils.py","file_url":"https://github.com/boschresearch/Open3DSG/blob/HEAD/open3dsg/models/pointnet2_utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"ec413739d406e611"}},{"code_sha256_prefix":"b5886a512bef5688","entry":"rotation_matrix","repo":"boschresearch/Open3DSG","repo_kind":"official","path":"open3dsg/models/op_utils.py","file_url":"https://github.com/boschresearch/Open3DSG/blob/HEAD/open3dsg/models/op_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"b5886a512bef5688"}},{"code_sha256_prefix":"b716f990583c288b","entry":"rotation_matrix_from_vectors","repo":"boschresearch/Open3DSG","repo_kind":"official","path":"open3dsg/models/op_utils.py","file_url":"https://github.com/boschresearch/Open3DSG/blob/HEAD/open3dsg/models/op_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"b716f990583c288b"}},{"code_sha256_prefix":"580dee6761649b29","entry":"square_distance","repo":"boschresearch/Open3DSG","repo_kind":"official","path":"open3dsg/models/pointnet2_utils.py","file_url":"https://github.com/boschresearch/Open3DSG/blob/HEAD/open3dsg/models/pointnet2_utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"580dee6761649b29"}},{"code_sha256_prefix":"ad8ae08946409876","entry":"feature_transform_reguliarzer","repo":"boschresearch/Open3DSG","repo_kind":"official","path":"open3dsg/models/pointnet.py","file_url":"https://github.com/boschresearch/Open3DSG/blob/HEAD/open3dsg/models/pointnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"ad8ae08946409876"}},{"code_sha256_prefix":"b1227ddb721e2999","entry":"timeit","repo":"boschresearch/Open3DSG","repo_kind":"official","path":"open3dsg/models/pointnet2_utils.py","file_url":"https://github.com/boschresearch/Open3DSG/blob/HEAD/open3dsg/models/pointnet2_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"AGPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"b1227ddb721e2999"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}