{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/opendlign-enhancing-open-world-3d-learning","title":"OpenDlign: Open-World Point Cloud Understanding with Depth-Aligned Images","arxiv_id":"2404.16538","date":"2024-04-25","proceeding":null,"authors":["Ye Mao","Junpeng Jing","Krystian Mikolajczyk"],"abstract":"Recent open-world 3D representation learning methods using Vision-Language Models (VLMs) to align 3D point cloud with image-text information have shown superior 3D zero-shot performance. However, CAD-rendered images for this alignment often lack realism and texture variation, compromising alignment robustness. Moreover, the volume discrepancy between 3D and 2D pretraining datasets highlights the need for effective strategies to transfer the representational abilities of VLMs to 3D learning. In this paper, we present OpenDlign, a novel open-world 3D model using depth-aligned images generated from a diffusion model for robust multimodal alignment. These images exhibit greater texture diversity than CAD renderings due to the stochastic nature of the diffusion model. By refining the depth map projection pipeline and designing depth-specific prompts, OpenDlign leverages rich knowledge in pre-trained VLM for 3D representation learning with streamlined fine-tuning. Our experiments show that OpenDlign achieves high zero-shot and few-shot performance on diverse 3D tasks, despite only fine-tuning 6 million parameters on a limited ShapeNet dataset. In zero-shot classification, OpenDlign surpasses previous models by 8.0% on ModelNet40 and 16.4% on OmniObject3D. Additionally, using depth-aligned images for multimodal alignment consistently enhances the performance of other state-of-the-art models.","url_abs":"https://arxiv.org/abs/2404.16538v3","url_pdf":"https://arxiv.org/pdf/2404.16538v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"opendlign-enhancing-open-world-3d-learning","repo_url":"https://github.com/Yebulabula/OpenDlign","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"},{"task_slug":"zero-shot-transfer-3d-point-cloud","task_name":"Zero-Shot Transfer 3D Point Cloud Classification"},{"task_slug":"zero-shot-3d-point-cloud-classification","task_name":"Zero-shot 3D Point Cloud Classification"},{"task_slug":"zero-shot-3d-classification","task_name":"Zero-shot 3D classification"},{"task_slug":null,"task_name":"zero-shot-classification"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/zero-shot-transfer-3d-point-cloud","task":"Zero-Shot Transfer 3D Point Cloud Classification","dataset":"ModelNet40","model":"TAMM-PointBERT (+dlign)","rank_in_archive_order":5,"of":16,"metrics":{"Accuracy (%)":"86.2"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-transfer-3d-point-cloud","task":"Zero-Shot Transfer 3D Point Cloud Classification","dataset":"ModelNet40","model":"OpenShape-PointBERT (+dlign)","rank_in_archive_order":6,"of":16,"metrics":{"Accuracy (%)":"85.4"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-transfer-3d-point-cloud","task":"Zero-Shot Transfer 3D Point Cloud Classification","dataset":"ModelNet40","model":"OpenShape-SparseConv (+dlign)","rank_in_archive_order":8,"of":16,"metrics":{"Accuracy (%)":"85.0"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-transfer-3d-point-cloud","task":"Zero-Shot Transfer 3D Point Cloud Classification","dataset":"ModelNet40","model":"OpenDlign","rank_in_archive_order":10,"of":16,"metrics":{"Accuracy (%)":"82.6"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-transfer-3d-point-cloud-2","task":"Zero-Shot Transfer 3D Point Cloud Classification","dataset":"ScanObjectNN","model":"TAMM-PointBERT (+dlign)","rank_in_archive_order":3,"of":10,"metrics":{"OBJ_ONLY Accuracy(%)":"60.5"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-transfer-3d-point-cloud-2","task":"Zero-Shot Transfer 3D Point Cloud Classification","dataset":"ScanObjectNN","model":"OpenDlign","rank_in_archive_order":5,"of":10,"metrics":{"OBJ_ONLY Accuracy(%)":"59.5"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.16538","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.16538"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Yebulabula/OpenDlign","reach":null}],"summary":{"ran":1,"ran_honours":1,"ran_draft_wrong":1,"ran_fixture":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"d328e829f7e326f8","entry":"OpenDlign","repo":"Yebulabula/OpenDlign","repo_kind":"official","path":"model.py","file_url":"https://github.com/Yebulabula/OpenDlign/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d328e829f7e326f8"}},{"code_sha256_prefix":"b2947808596ad0ac","entry":"get_num_classes","repo":"Yebulabula/OpenDlign","repo_kind":"official","path":"zero_shot.py","file_url":"https://github.com/Yebulabula/OpenDlign/blob/HEAD/zero_shot.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b2947808596ad0ac"}},{"code_sha256_prefix":"6476c74ec6b02b24","entry":"initialize_metrics","repo":"Yebulabula/OpenDlign","repo_kind":"official","path":"zero_shot.py","file_url":"https://github.com/Yebulabula/OpenDlign/blob/HEAD/zero_shot.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6476c74ec6b02b24"}},{"code_sha256_prefix":"bef35f2826433466","entry":"process_text_features","repo":"Yebulabula/OpenDlign","repo_kind":"official","path":"zero_shot.py","file_url":"https://github.com/Yebulabula/OpenDlign/blob/HEAD/zero_shot.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bef35f2826433466"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}