{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multimodality-helps-few-shot-3d-point-cloud","title":"Multimodality Helps Few-Shot 3D Point Cloud Semantic Segmentation","arxiv_id":"2410.22489","date":"2024-10-29","proceeding":null,"authors":["Zhaochong An","Guolei Sun","Yun Liu","Runjia Li","Min Wu","Ming-Ming Cheng","Ender Konukoglu","Serge Belongie"],"abstract":"Few-shot 3D point cloud segmentation (FS-PCS) aims at generalizing models to segment novel categories with minimal annotated support samples. While existing FS-PCS methods have shown promise, they primarily focus on unimodal point cloud inputs, overlooking the potential benefits of leveraging multimodal information. In this paper, we address this gap by introducing a cost-free multimodal FS-PCS setup, utilizing textual labels and the potentially available 2D image modality. Under this easy-to-achieve setup, we present the MultiModal Few-Shot SegNet (MM-FSS), a model effectively harnessing complementary information from multiple modalities. MM-FSS employs a shared backbone with two heads to extract intermodal and unimodal visual features, and a pretrained text encoder to generate text embeddings. To fully exploit the multimodal information, we propose a Multimodal Correlation Fusion (MCF) module to generate multimodal correlations, and a Multimodal Semantic Fusion (MSF) module to refine the correlations using text-aware semantic guidance. Additionally, we propose a simple yet effective Test-time Adaptive Cross-modal Calibration (TACC) technique to mitigate training bias, further improving generalization. Experimental results on S3DIS and ScanNet datasets demonstrate significant performance improvements achieved by our method. The efficacy of our approach indicates the benefits of leveraging commonly-ignored free modalities for FS-PCS, providing valuable insights for future research. The code is available at https://github.com/ZhaochongAn/Multimodality-3D-Few-Shot","url_abs":"https://arxiv.org/abs/2410.22489v2","url_pdf":"https://arxiv.org/pdf/2410.22489v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multimodality-helps-few-shot-3d-point-cloud","repo_url":"https://github.com/zhaochongan/multimodality-3d-few-shot","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"multimodality-helps-few-shot-3d-point-cloud","repo_url":"https://github.com/zhaochongan/coseg","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"few-shot-3d-point-cloud-semantic-segmentation","task_name":"Few-shot 3D Point Cloud Semantic Segmentation"},{"task_slug":"point-cloud-segmentation","task_name":"Point Cloud Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"kaiming-initialization","method_name":"Kaiming Initialization"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"segnet","method_name":"SegNet"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.22489","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.22489"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhaochongan/multimodality-3d-few-shot","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhaochongan/coseg","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"1c650e412643685a","entry":"evaluate_metric","repo":"zhaochongan/multimodality-3d-few-shot","repo_kind":"official","path":"util/common_util.py","file_url":"https://github.com/zhaochongan/multimodality-3d-few-shot/blob/HEAD/util/common_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1c650e412643685a"}},{"code_sha256_prefix":"a8a8b079abda7437","entry":"intersectionAndUnion","repo":"zhaochongan/multimodality-3d-few-shot","repo_kind":"official","path":"util/common_util.py","file_url":"https://github.com/zhaochongan/multimodality-3d-few-shot/blob/HEAD/util/common_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a8a8b079abda7437"}},{"code_sha256_prefix":"7e5e0eaaecfc467e","entry":"load_pretrain_checkpoint","repo":"zhaochongan/multimodality-3d-few-shot","repo_kind":"official","path":"util/common_util.py","file_url":"https://github.com/zhaochongan/multimodality-3d-few-shot/blob/HEAD/util/common_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7e5e0eaaecfc467e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}