{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-unified-framework-for-3d-scene","title":"A Unified Framework for 3D Scene Understanding","arxiv_id":"2407.03263","date":"2024-07-03","proceeding":null,"authors":["Wei Xu","Chunsheng Shi","Sifan Tu","Xin Zhou","Dingkang Liang","Xiang Bai"],"abstract":"We propose UniSeg3D, a unified 3D scene understanding framework that achieves panoptic, semantic, instance, interactive, referring, and open-vocabulary segmentation tasks within a single model. Most previous 3D segmentation approaches are typically tailored to a specific task, limiting their understanding of 3D scenes to a task-specific perspective. In contrast, the proposed method unifies six tasks into unified representations processed by the same Transformer. It facilitates inter-task knowledge sharing, thereby promoting comprehensive 3D scene understanding. To take advantage of multi-task unification, we enhance performance by establishing explicit inter-task associations. Specifically, we design knowledge distillation and contrastive learning methods to transfer task-specific knowledge across different tasks. Experiments on three benchmarks, including ScanNet20, ScanRefer, and ScanNet200, demonstrate that the UniSeg3D consistently outperforms current SOTA methods, even those specialized for individual tasks. We hope UniSeg3D can serve as a solid unified baseline and inspire future work. Code and models are available at https://github.com/dk-liang/UniSeg3D.","url_abs":"https://arxiv.org/abs/2407.03263v2","url_pdf":"https://arxiv.org/pdf/2407.03263v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-unified-framework-for-3d-scene","repo_url":"https://github.com/dk-liang/uniseg3d","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"open-vocabulary-semantic-segmentation","task_name":"Open Vocabulary Semantic Segmentation"},{"task_slug":"open-vocabulary-semantic-segmentation-1","task_name":"Open-Vocabulary Semantic Segmentation"},{"task_slug":"scene-understanding","task_name":"Scene Understanding"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.03263","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.03263"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dk-liang/uniseg3d","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":7},"by_repo_kind":{"official":{"samples":7,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"cdbb52ae3380c642","entry":"aggregate_predictions","repo":"dk-liang/uniseg3d","repo_kind":"official","path":"uniseg3d/instance_seg_eval.py","file_url":"https://github.com/dk-liang/uniseg3d/blob/HEAD/uniseg3d/instance_seg_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cdbb52ae3380c642"}},{"code_sha256_prefix":"466d41757b35f3b7","entry":"batch_dice_loss","repo":"dk-liang/uniseg3d","repo_kind":"official","path":"uniseg3d/instance_criterion.py","file_url":"https://github.com/dk-liang/uniseg3d/blob/HEAD/uniseg3d/instance_criterion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"466d41757b35f3b7"}},{"code_sha256_prefix":"bb90a7c29aefe4b5","entry":"batch_sigmoid_bce_loss","repo":"dk-liang/uniseg3d","repo_kind":"official","path":"uniseg3d/instance_criterion.py","file_url":"https://github.com/dk-liang/uniseg3d/blob/HEAD/uniseg3d/instance_criterion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bb90a7c29aefe4b5"}},{"code_sha256_prefix":"b0c2c75dd7b3bea7","entry":"compute_averages","repo":"dk-liang/uniseg3d","repo_kind":"official","path":"uniseg3d/evaluate_semantic_instance.py","file_url":"https://github.com/dk-liang/uniseg3d/blob/HEAD/uniseg3d/evaluate_semantic_instance.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b0c2c75dd7b3bea7"}},{"code_sha256_prefix":"406a0a541539fd95","entry":"evaluate_matches","repo":"dk-liang/uniseg3d","repo_kind":"official","path":"uniseg3d/evaluate_semantic_instance.py","file_url":"https://github.com/dk-liang/uniseg3d/blob/HEAD/uniseg3d/evaluate_semantic_instance.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"406a0a541539fd95"}},{"code_sha256_prefix":"733b1658e2226813","entry":"get_iou","repo":"dk-liang/uniseg3d","repo_kind":"official","path":"uniseg3d/instance_criterion.py","file_url":"https://github.com/dk-liang/uniseg3d/blob/HEAD/uniseg3d/instance_criterion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"733b1658e2226813"}},{"code_sha256_prefix":"1f18a102cc1b1c23","entry":"rename_gt","repo":"dk-liang/uniseg3d","repo_kind":"official","path":"uniseg3d/instance_seg_eval.py","file_url":"https://github.com/dk-liang/uniseg3d/blob/HEAD/uniseg3d/instance_seg_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1f18a102cc1b1c23"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}