{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/open-vocabulary-3d-scene-understanding-via","title":"Open Vocabulary 3D Scene Understanding via Geometry Guided Self-Distillation","arxiv_id":"2407.13362","date":"2024-07-18","proceeding":null,"authors":["Pengfei Wang","Yuxi Wang","Shuai Li","Zhaoxiang Zhang","Zhen Lei","Lei Zhang"],"abstract":"The scarcity of large-scale 3D-text paired data poses a great challenge on open vocabulary 3D scene understanding, and hence it is popular to leverage internet-scale 2D data and transfer their open vocabulary capabilities to 3D models through knowledge distillation. However, the existing distillation-based 3D scene understanding approaches rely on the representation capacity of 2D models, disregarding the exploration of geometric priors and inherent representational advantages offered by 3D data. In this paper, we propose an effective approach, namely Geometry Guided Self-Distillation (GGSD), to learn superior 3D representations from 2D pre-trained models. Specifically, we first design a geometry guided distillation module to distill knowledge from 2D models, and then leverage the 3D geometric priors to alleviate the inherent noise in 2D models and enhance the representation learning process. Due to the advantages of 3D representation, the performance of the distilled 3D student model can significantly surpass that of the 2D teacher model. This motivates us to further leverage the representation advantages of 3D data through self-distillation. As a result, our proposed GGSD approach outperforms the existing open vocabulary 3D scene understanding methods by a large margin, as demonstrated by our experiments on both indoor and outdoor benchmark datasets.","url_abs":"https://arxiv.org/abs/2407.13362v1","url_pdf":"https://arxiv.org/pdf/2407.13362v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"scene-understanding","task_name":"Scene Understanding"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2407.13362","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.13362"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Wang-pengfei/GGSD","reach":{"status":"ok"}}],"summary":{"ran":4,"unverified":5},"by_repo_kind":{"found_in_text":{"samples":9,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"f726974066cf1352","entry":"collation_fn","repo":"Wang-pengfei/GGSD","repo_kind":"found_in_text","path":"dataset/feature_loader.py","file_url":"https://github.com/Wang-pengfei/GGSD/blob/HEAD/dataset/feature_loader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f726974066cf1352"}},{"code_sha256_prefix":"65392c9cae62b77b","entry":"collation_fn_eval_all","repo":"Wang-pengfei/GGSD","repo_kind":"found_in_text","path":"dataset/feature_loader.py","file_url":"https://github.com/Wang-pengfei/GGSD/blob/HEAD/dataset/feature_loader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"65392c9cae62b77b"}},{"code_sha256_prefix":"cc9a3557465a7813","entry":"collation_fn_eval_all","repo":"Wang-pengfei/GGSD","repo_kind":"found_in_text","path":"dataset/point_loader.py","file_url":"https://github.com/Wang-pengfei/GGSD/blob/HEAD/dataset/point_loader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cc9a3557465a7813"}},{"code_sha256_prefix":"c4b88bc6388a332e","entry":"state_dict_remove_moudle","repo":"Wang-pengfei/GGSD","repo_kind":"found_in_text","path":"models/disnet.py","file_url":"https://github.com/Wang-pengfei/GGSD/blob/HEAD/models/disnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c4b88bc6388a332e"}},{"code_sha256_prefix":"3095f89bf0fc823d","entry":"M","repo":"Wang-pengfei/GGSD","repo_kind":"found_in_text","path":"dataset/voxelizer.py","file_url":"https://github.com/Wang-pengfei/GGSD/blob/HEAD/dataset/voxelizer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3095f89bf0fc823d"}},{"code_sha256_prefix":"bc5616d55b1b06b2","entry":"collation_fn","repo":"Wang-pengfei/GGSD","repo_kind":"found_in_text","path":"dataset/point_loader.py","file_url":"https://github.com/Wang-pengfei/GGSD/blob/HEAD/dataset/point_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bc5616d55b1b06b2"}},{"code_sha256_prefix":"0086c892bd1df796","entry":"fnv_hash_vec","repo":"Wang-pengfei/GGSD","repo_kind":"found_in_text","path":"dataset/voxelization_utils.py","file_url":"https://github.com/Wang-pengfei/GGSD/blob/HEAD/dataset/voxelization_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0086c892bd1df796"}},{"code_sha256_prefix":"5905d8e45b03578e","entry":"get_release_scans","repo":"Wang-pengfei/GGSD","repo_kind":"found_in_text","path":"dataset/scannet/download-scannet.py","file_url":"https://github.com/Wang-pengfei/GGSD/blob/HEAD/dataset/scannet/download-scannet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5905d8e45b03578e"}},{"code_sha256_prefix":"afdc8ad15d9098b8","entry":"ravel_hash_vec","repo":"Wang-pengfei/GGSD","repo_kind":"found_in_text","path":"dataset/voxelization_utils.py","file_url":"https://github.com/Wang-pengfei/GGSD/blob/HEAD/dataset/voxelization_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"afdc8ad15d9098b8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}