{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rethinking-end-to-end-2d-to-3d-scene","title":"Rethinking End-to-End 2D to 3D Scene Segmentation in Gaussian Splatting","arxiv_id":"2503.14029","date":"2025-03-18","proceeding":"CVPR 2025 1","authors":["Runsong Zhu","Shi Qiu","Zhengzhe Liu","Ka-Hei Hui","Qianyi Wu","Pheng-Ann Heng","Chi-Wing Fu"],"abstract":"Lifting multi-view 2D instance segmentation to a radiance field has proven to be effective to enhance 3D understanding. Existing methods rely on direct matching for end-to-end lifting, yielding inferior results; or employ a two-stage solution constrained by complex pre- or post-processing. In this work, we design a new end-to-end object-aware lifting approach, named Unified-Lift that provides accurate 3D segmentation based on the 3D Gaussian representation. To start, we augment each Gaussian point with an additional Gaussian-level feature learned using a contrastive loss to encode instance information. Importantly, we introduce a learnable object-level codebook to account for individual objects in the scene for an explicit object-level understanding and associate the encoded object-level features with the Gaussian-level point features for segmentation predictions. While promising, achieving effective codebook learning is non-trivial and a naive solution leads to degraded performance. Therefore, we formulate the association learning module and the noisy label filtering module for effective and robust codebook learning. We conduct experiments on three benchmarks: LERF-Masked, Replica, and Messy Rooms datasets. Both qualitative and quantitative results manifest that our Unified-Lift clearly outperforms existing methods in terms of segmentation quality and time efficiency. The code is publicly available at \\href{https://github.com/Runsong123/Unified-Lift}{https://github.com/Runsong123/Unified-Lift}.","url_abs":"https://arxiv.org/abs/2503.14029v1","url_pdf":"https://arxiv.org/pdf/2503.14029v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rethinking-end-to-end-2d-to-3d-scene","repo_url":"https://github.com/runsong123/unified-lift","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"object","task_name":"Object"},{"task_slug":"scene-segmentation","task_name":"Scene Segmentation"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2503.14029","atlas_url":"https://app.syntology.ai/?focus=2503.14029","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.14029"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/runsong123/unified-lift","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":5,"unverified":6},"by_repo_kind":{"official":{"samples":11,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c20005191e0618bd","entry":"boundary_iou","repo":"runsong123/unified-lift","repo_kind":"official","path":"script/eval_lerf_mask_unified_lift.py","file_url":"https://github.com/runsong123/unified-lift/blob/HEAD/script/eval_lerf_mask_unified_lift.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c20005191e0618bd"}},{"code_sha256_prefix":"5f91cd17bff041c5","entry":"get_confience_map","repo":"runsong123/unified-lift","repo_kind":"official","path":"train_unified_lift.py","file_url":"https://github.com/runsong123/unified-lift/blob/HEAD/train_unified_lift.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5f91cd17bff041c5"}},{"code_sha256_prefix":"ac0e42d6fbcfbbe6","entry":"l1_loss","repo":"runsong123/unified-lift","repo_kind":"official","path":"utils/loss_utils.py","file_url":"https://github.com/runsong123/unified-lift/blob/HEAD/utils/loss_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ac0e42d6fbcfbbe6"}},{"code_sha256_prefix":"bf2d28223bdf332b","entry":"load_mask","repo":"runsong123/unified-lift","repo_kind":"official","path":"script/eval_lerf_mask_unified_lift.py","file_url":"https://github.com/runsong123/unified-lift/blob/HEAD/script/eval_lerf_mask_unified_lift.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bf2d28223bdf332b"}},{"code_sha256_prefix":"b79a90dcf06de145","entry":"mask_to_boundary","repo":"runsong123/unified-lift","repo_kind":"official","path":"script/eval_lerf_mask_unified_lift.py","file_url":"https://github.com/runsong123/unified-lift/blob/HEAD/script/eval_lerf_mask_unified_lift.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b79a90dcf06de145"}},{"code_sha256_prefix":"955c607d3685039a","entry":"feature_to_rgb","repo":"runsong123/unified-lift","repo_kind":"official","path":"render.py","file_url":"https://github.com/runsong123/unified-lift/blob/HEAD/render.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"955c607d3685039a"}},{"code_sha256_prefix":"9acdd81f4e30bd46","entry":"id2rgb","repo":"runsong123/unified-lift","repo_kind":"official","path":"train_unified_lift.py","file_url":"https://github.com/runsong123/unified-lift/blob/HEAD/train_unified_lift.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9acdd81f4e30bd46"}},{"code_sha256_prefix":"0ac7968bd7da45d3","entry":"masked_l1_loss","repo":"runsong123/unified-lift","repo_kind":"official","path":"utils/loss_utils.py","file_url":"https://github.com/runsong123/unified-lift/blob/HEAD/utils/loss_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0ac7968bd7da45d3"}},{"code_sha256_prefix":"eb026ce356b1b155","entry":"show_mask","repo":"runsong123/unified-lift","repo_kind":"official","path":"ext/grounded_sam.py","file_url":"https://github.com/runsong123/unified-lift/blob/HEAD/ext/grounded_sam.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eb026ce356b1b155"}},{"code_sha256_prefix":"74195777dbe870e4","entry":"visualize_obj","repo":"runsong123/unified-lift","repo_kind":"official","path":"train_unified_lift.py","file_url":"https://github.com/runsong123/unified-lift/blob/HEAD/train_unified_lift.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"74195777dbe870e4"}},{"code_sha256_prefix":"e14045a7568f1af5","entry":"weighted_l1_loss","repo":"runsong123/unified-lift","repo_kind":"official","path":"utils/loss_utils.py","file_url":"https://github.com/runsong123/unified-lift/blob/HEAD/utils/loss_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e14045a7568f1af5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}