{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/imov3d-learning-open-vocabulary-point-clouds","title":"ImOV3D: Learning Open-Vocabulary Point Clouds 3D Object Detection from Only 2D Images","arxiv_id":"2410.24001","date":"2024-10-31","proceeding":null,"authors":["Timing Yang","Yuanliang Ju","Li Yi"],"abstract":"Open-vocabulary 3D object detection (OV-3Det) aims to generalize beyond the limited number of base categories labeled during the training phase. The biggest bottleneck is the scarcity of annotated 3D data, whereas 2D image datasets are abundant and richly annotated. Consequently, it is intuitive to leverage the wealth of annotations in 2D images to alleviate the inherent data scarcity in OV-3Det. In this paper, we push the task setup to its limits by exploring the potential of using solely 2D images to learn OV-3Det. The major challenges for this setup is the modality gap between training images and testing point clouds, which prevents effective integration of 2D knowledge into OV-3Det. To address this challenge, we propose a novel framework ImOV3D to leverage pseudo multimodal representation containing both images and point clouds (PC) to close the modality gap. The key of ImOV3D lies in flexible modality conversion where 2D images can be lifted into 3D using monocular depth estimation and can also be derived from 3D scenes through rendering. This allows unifying both training images and testing point clouds into a common image-PC representation, encompassing a wealth of 2D semantic information and also incorporating the depth and structural characteristics of 3D spatial data. We carefully conduct such conversion to minimize the domain gap between training and test cases. Extensive experiments on two benchmark datasets, SUNRGBD and ScanNet, show that ImOV3D significantly outperforms existing methods, even in the absence of ground truth 3D training data. With the inclusion of a minimal amount of real 3D data for fine-tuning, the performance also significantly surpasses previous state-of-the-art. Codes and pre-trained models are released on the https://github.com/yangtiming/ImOV3D.","url_abs":"https://arxiv.org/abs/2410.24001v1","url_pdf":"https://arxiv.org/pdf/2410.24001v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"imov3d-learning-open-vocabulary-point-clouds","repo_url":"https://github.com/yangtiming/imov3d","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"3d-object-detection","task_name":"3D Object Detection"},{"task_slug":"depth-estimation","task_name":"Depth Estimation"},{"task_slug":"monocular-depth-estimation","task_name":"Monocular Depth Estimation"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"base","method_name":"BASE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.24001","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.24001"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/yangtiming/ImOV3D","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yangtiming/imov3d","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":1,"ran_draft_wrong":1,"unverified":5},"by_repo_kind":{"official":{"samples":7,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"eddd23cbbb297813","entry":"flip_axis_to_camera","repo":"yangtiming/ImOV3D","repo_kind":"official","path":"models/ap_helper.py","file_url":"https://github.com/yangtiming/ImOV3D/blob/HEAD/models/ap_helper.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eddd23cbbb297813"}},{"code_sha256_prefix":"847f37b80100244c","entry":"sample_valid_seeds","repo":"yangtiming/ImOV3D","repo_kind":"official","path":"models/imvotenet.py","file_url":"https://github.com/yangtiming/ImOV3D/blob/HEAD/models/imvotenet.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"847f37b80100244c"}},{"code_sha256_prefix":"c0e66b3130d33e86","entry":"append_img_feat","repo":"yangtiming/ImOV3D","repo_kind":"official","path":"models/image_feature_module.py","file_url":"https://github.com/yangtiming/ImOV3D/blob/HEAD/models/image_feature_module.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c0e66b3130d33e86"}},{"code_sha256_prefix":"5511eb60d27d2fed","entry":"decode_scores","repo":"yangtiming/ImOV3D","repo_kind":"official","path":"models/proposal_module.py","file_url":"https://github.com/yangtiming/ImOV3D/blob/HEAD/models/proposal_module.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5511eb60d27d2fed"}},{"code_sha256_prefix":"d638b0201b9a9ece","entry":"flip_axis_to_depth","repo":"yangtiming/ImOV3D","repo_kind":"official","path":"models/ap_helper.py","file_url":"https://github.com/yangtiming/ImOV3D/blob/HEAD/models/ap_helper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d638b0201b9a9ece"}},{"code_sha256_prefix":"577fe993346effd2","entry":"get_flags","repo":"yangtiming/ImOV3D","repo_kind":"official","path":"config.py","file_url":"https://github.com/yangtiming/ImOV3D/blob/HEAD/config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"577fe993346effd2"}},{"code_sha256_prefix":"1a034a72c15f9903","entry":"softmax","repo":"yangtiming/ImOV3D","repo_kind":"official","path":"models/ap_helper.py","file_url":"https://github.com/yangtiming/ImOV3D/blob/HEAD/models/ap_helper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1a034a72c15f9903"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}