{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ulip-2-towards-scalable-multimodal-pre","title":"ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding","arxiv_id":"2305.08275","date":"2023-05-14","proceeding":"CVPR 2024 1","authors":["Le Xue","Ning Yu","Shu Zhang","Artemis Panagopoulou","Junnan Li","Roberto Martín-Martín","Jiajun Wu","Caiming Xiong","ran Xu","Juan Carlos Niebles","Silvio Savarese"],"abstract":"Recent advancements in multimodal pre-training have shown promising efficacy in 3D representation learning by aligning multimodal features across 3D shapes, their 2D counterparts, and language descriptions. However, the methods used by existing frameworks to curate such multimodal data, in particular language descriptions for 3D shapes, are not scalable, and the collected language descriptions are not diverse. To address this, we introduce ULIP-2, a simple yet effective tri-modal pre-training framework that leverages large multimodal models to automatically generate holistic language descriptions for 3D shapes. It only needs 3D data as input, eliminating the need for any manual 3D annotations, and is therefore scalable to large datasets. ULIP-2 is also equipped with scaled-up backbones for better multimodal representation learning. We conduct experiments on two large-scale 3D datasets, Objaverse and ShapeNet, and augment them with tri-modal datasets of 3D point clouds, images, and language for training ULIP-2. Experiments show that ULIP-2 demonstrates substantial benefits in three downstream tasks: zero-shot 3D classification, standard 3D classification with fine-tuning, and 3D captioning (3D-to-language generation). It achieves a new SOTA of 50.6% (top-1) on Objaverse-LVIS and 84.7% (top-1) on ModelNet40 in zero-shot classification. In the ScanObjectNN benchmark for standard fine-tuning, ULIP-2 reaches an overall accuracy of 91.5% with a compact model of only 1.4 million parameters. ULIP-2 sheds light on a new paradigm for scalable multimodal 3D representation learning without human annotations and shows significant improvements over existing baselines. The code and datasets are released at https://github.com/salesforce/ULIP.","url_abs":"https://arxiv.org/abs/2305.08275v4","url_pdf":"https://arxiv.org/pdf/2305.08275v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ulip-2-towards-scalable-multimodal-pre","repo_url":"https://github.com/salesforce/ulip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"3d-classification","task_name":"3D Classification"},{"task_slug":"3d-point-cloud-classification","task_name":"3D Point Cloud Classification"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"},{"task_slug":"zero-shot-3d-classification","task_name":"Zero-shot 3D classification"},{"task_slug":null,"task_name":"zero-shot-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/3d-point-cloud-classification-on-scanobjectnn","task":"3D Point Cloud Classification","dataset":"ScanObjectNN","model":"ULIP-2 + PointNeXt","rank_in_archive_order":11,"of":77,"metrics":{"Mean Accuracy":"91.2","Number of params":"1.4M","Overall Accuracy":"91.5"},"uses_additional_data":true},{"leaderboard":"/sota/3d-point-cloud-classification-on-scanobjectnn","task":"3D Point Cloud Classification","dataset":"ScanObjectNN","model":"ULIP-2 + PointNeXt (no voting)","rank_in_archive_order":13,"of":77,"metrics":{"Mean Accuracy":"90.3","Number of params":"1.4M","Overall Accuracy":"90.8"},"uses_additional_data":true},{"leaderboard":"/sota/3d-point-cloud-classification-on-scanobjectnn","task":"3D Point Cloud Classification","dataset":"ScanObjectNN","model":"ULIP-2 + Point-BERT","rank_in_archive_order":32,"of":77,"metrics":{"Overall Accuracy":"89.0"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.08275","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.08275"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/salesforce/ulip","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"ran_draft_wrong":1},"by_repo_kind":{},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"01955600041e2b19","entry":"get_metric_names","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"01955600041e2b19"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}