{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pimae-point-cloud-and-image-interactive","title":"PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detection","arxiv_id":"2303.08129","date":"2023-03-14","proceeding":"CVPR 2023 1","authors":["Anthony Chen","Kevin Zhang","Renrui Zhang","Zihan Wang","Yuheng Lu","Yandong Guo","Shanghang Zhang"],"abstract":"Masked Autoencoders learn strong visual representations and achieve state-of-the-art results in several independent modalities, yet very few works have addressed their capabilities in multi-modality settings. In this work, we focus on point cloud and RGB image data, two modalities that are often presented together in the real world, and explore their meaningful interactions. To improve upon the cross-modal synergy in existing works, we propose PiMAE, a self-supervised pre-training framework that promotes 3D and 2D interaction through three aspects. Specifically, we first notice the importance of masking strategies between the two sources and utilize a projection module to complementarily align the mask and visible tokens of the two modalities. Then, we utilize a well-crafted two-branch MAE pipeline with a novel shared decoder to promote cross-modality interaction in the mask tokens. Finally, we design a unique cross-modal reconstruction module to enhance representation learning for both modalities. Through extensive experiments performed on large-scale RGB-D scene understanding benchmarks (SUN RGB-D and ScannetV2), we discover it is nontrivial to interactively learn point-image features, where we greatly improve multiple 3D detectors, 2D detectors, and few-shot classifiers by 2.9%, 6.7%, and 2.4%, respectively. Code is available at https://github.com/BLVLab/PiMAE.","url_abs":"https://arxiv.org/abs/2303.08129v1","url_pdf":"https://arxiv.org/pdf/2303.08129v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pimae-point-cloud-and-image-interactive","repo_url":"https://github.com/blvlab/pimae","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"3d-object-detection","task_name":"3D Object Detection"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"scene-understanding","task_name":"Scene Understanding"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"mae","method_name":"MAE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2303.08129","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.08129"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/blvlab/pimae","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/BLVLab/PiMAE","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"130bf18af7f1afde","entry":"BatchNormDim1Swap","repo":"BLVLab/PiMAE","repo_kind":"official","path":"Pretrain/models/pimae.py","file_url":"https://github.com/BLVLab/PiMAE/blob/HEAD/Pretrain/models/pimae.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"130bf18af7f1afde"}},{"code_sha256_prefix":"fd6e69429d55cc68","entry":"Block","repo":"BLVLab/PiMAE","repo_kind":"official","path":"Pretrain/models/pimae.py","file_url":"https://github.com/BLVLab/PiMAE/blob/HEAD/Pretrain/models/pimae.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fd6e69429d55cc68"}},{"code_sha256_prefix":"81da177eaf52ee01","entry":"DropPath","repo":"BLVLab/PiMAE","repo_kind":"official","path":"Pretrain/models/pimae.py","file_url":"https://github.com/BLVLab/PiMAE/blob/HEAD/Pretrain/models/pimae.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"81da177eaf52ee01"}},{"code_sha256_prefix":"1d16864030bea18b","entry":"Attention","repo":"BLVLab/PiMAE","repo_kind":"official","path":"Pretrain/models/pimae.py","file_url":"https://github.com/BLVLab/PiMAE/blob/HEAD/Pretrain/models/pimae.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1d16864030bea18b"}},{"code_sha256_prefix":"b02ff5d4bf504040","entry":"PiMAE","repo":"BLVLab/PiMAE","repo_kind":"official","path":"Pretrain/models/pimae.py","file_url":"https://github.com/BLVLab/PiMAE/blob/HEAD/Pretrain/models/pimae.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b02ff5d4bf504040"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}