{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/xmask3d-cross-modal-mask-reasoning-for-open","title":"XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentation","arxiv_id":"2411.13243","date":"2024-11-20","proceeding":null,"authors":["Ziyi Wang","Yanbo Wang","Xumin Yu","Jie zhou","Jiwen Lu"],"abstract":"Existing methodologies in open vocabulary 3D semantic segmentation primarily concentrate on establishing a unified feature space encompassing 3D, 2D, and textual modalities. Nevertheless, traditional techniques such as global feature alignment or vision-language model distillation tend to impose only approximate correspondence, struggling notably with delineating fine-grained segmentation boundaries. To address this gap, we propose a more meticulous mask-level alignment between 3D features and the 2D-text embedding space through a cross-modal mask reasoning framework, XMask3D. In our approach, we developed a mask generator based on the denoising UNet from a pre-trained diffusion model, leveraging its capability for precise textual control over dense pixel representations and enhancing the open-world adaptability of the generated masks. We further integrate 3D global features as implicit conditions into the pre-trained 2D denoising UNet, enabling the generation of segmentation masks with additional 3D geometry awareness. Subsequently, the generated 2D masks are employed to align mask-level 3D representations with the vision-language feature space, thereby augmenting the open vocabulary capability of 3D geometry embeddings. Finally, we fuse complementary 2D and 3D mask features, resulting in competitive performance across multiple benchmarks for 3D open vocabulary semantic segmentation. Code is available at https://github.com/wangzy22/XMask3D.","url_abs":"https://arxiv.org/abs/2411.13243v1","url_pdf":"https://arxiv.org/pdf/2411.13243v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"xmask3d-cross-modal-mask-reasoning-for-open","repo_url":"https://github.com/wangzy22/xmask3d","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"3d-semantic-segmentation","task_name":"3D Semantic Segmentation"},{"task_slug":"3d-geometry","task_name":"3D geometry"},{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"open-vocabulary-semantic-segmentation","task_name":"Open Vocabulary Semantic Segmentation"},{"task_slug":"open-vocabulary-semantic-segmentation-1","task_name":"Open-Vocabulary Semantic Segmentation"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2411.13243","atlas_url":"https://app.syntology.ai/?focus=2411.13243","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.13243"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/wangzy22/XMask3D","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wangzy22/xmask3d","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":11},"by_repo_kind":{"official":{"samples":11,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3095f89bf0fc823d","entry":"M","repo":"wangzy22/XMask3D","repo_kind":"official","path":"dataset/voxelizer.py","file_url":"https://github.com/wangzy22/XMask3D/blob/HEAD/dataset/voxelizer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3095f89bf0fc823d"}},{"code_sha256_prefix":"69e60dbee942ddfa","entry":"batched_input_to_device","repo":"wangzy22/XMask3D","repo_kind":"official","path":"models/modeling/preprocess.py","file_url":"https://github.com/wangzy22/XMask3D/blob/HEAD/models/modeling/preprocess.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"69e60dbee942ddfa"}},{"code_sha256_prefix":"f3534330a66abdcf","entry":"collation_fn","repo":"wangzy22/XMask3D","repo_kind":"official","path":"dataset/data_loader.py","file_url":"https://github.com/wangzy22/XMask3D/blob/HEAD/dataset/data_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f3534330a66abdcf"}},{"code_sha256_prefix":"92cc05004dad14fc","entry":"collation_fn","repo":"wangzy22/XMask3D","repo_kind":"official","path":"dataset/feature_loader.py","file_url":"https://github.com/wangzy22/XMask3D/blob/HEAD/dataset/feature_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"92cc05004dad14fc"}},{"code_sha256_prefix":"323ca371ac90d3df","entry":"collation_fn","repo":"wangzy22/XMask3D","repo_kind":"official","path":"dataset/point_loader.py","file_url":"https://github.com/wangzy22/XMask3D/blob/HEAD/dataset/point_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"323ca371ac90d3df"}},{"code_sha256_prefix":"266fb124c37f5874","entry":"collation_fn_eval_all","repo":"wangzy22/XMask3D","repo_kind":"official","path":"dataset/feature_loader.py","file_url":"https://github.com/wangzy22/XMask3D/blob/HEAD/dataset/feature_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"266fb124c37f5874"}},{"code_sha256_prefix":"98877432b397d145","entry":"collation_fn_eval_all","repo":"wangzy22/XMask3D","repo_kind":"official","path":"dataset/point_loader.py","file_url":"https://github.com/wangzy22/XMask3D/blob/HEAD/dataset/point_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"98877432b397d145"}},{"code_sha256_prefix":"11e34c352ce9cd94","entry":"collation_fn_eval_all_full","repo":"wangzy22/XMask3D","repo_kind":"official","path":"dataset/data_loader_infer.py","file_url":"https://github.com/wangzy22/XMask3D/blob/HEAD/dataset/data_loader_infer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"11e34c352ce9cd94"}},{"code_sha256_prefix":"2f1cb6f962adbb70","entry":"fnv_hash_vec","repo":"wangzy22/XMask3D","repo_kind":"official","path":"dataset/voxelization_utils.py","file_url":"https://github.com/wangzy22/XMask3D/blob/HEAD/dataset/voxelization_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f1cb6f962adbb70"}},{"code_sha256_prefix":"b3e569142043c972","entry":"group_by_prefix","repo":"wangzy22/XMask3D","repo_kind":"official","path":"models/checkpoint/odise_checkpointer.py","file_url":"https://github.com/wangzy22/XMask3D/blob/HEAD/models/checkpoint/odise_checkpointer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b3e569142043c972"}},{"code_sha256_prefix":"9633e2a260e561fb","entry":"ravel_hash_vec","repo":"wangzy22/XMask3D","repo_kind":"official","path":"dataset/voxelization_utils.py","file_url":"https://github.com/wangzy22/XMask3D/blob/HEAD/dataset/voxelization_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9633e2a260e561fb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}