{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/self-supervised-learning-of-lidar-3d-point","title":"Self-supervised Learning of LiDAR 3D Point Clouds via 2D-3D Neural Calibration","arxiv_id":"2401.12452","date":"2024-01-23","proceeding":null,"authors":["Yifan Zhang","Siyu Ren","Junhui Hou","Jinjian Wu","Yixuan Yuan","Guangming Shi"],"abstract":"This paper introduces a novel self-supervised learning framework for enhancing 3D perception in autonomous driving scenes. Specifically, our approach, namely NCLR, focuses on 2D-3D neural calibration, a novel pretext task that estimates the rigid pose aligning camera and LiDAR coordinate systems. First, we propose the learnable transformation alignment to bridge the domain gap between image and point cloud data, converting features into a unified representation space for effective comparison and matching. Second, we identify the overlapping area between the image and point cloud with the fused features. Third, we establish dense 2D-3D correspondences to estimate the rigid pose. The framework not only learns fine-grained matching from points to pixels but also achieves alignment of the image and point cloud at a holistic level, understanding their relative pose. We demonstrate the efficacy of NCLR by applying the pre-trained backbone to downstream tasks, such as LiDAR-based 3D semantic segmentation, object detection, and panoptic segmentation. Comprehensive experiments on various datasets illustrate the superiority of NCLR over existing self-supervised methods. The results confirm that joint learning from different modalities significantly enhances the network's understanding abilities and effectiveness of learned representation. The code is publicly available at https://github.com/Eaphan/NCLR.","url_abs":"https://arxiv.org/abs/2401.12452v3","url_pdf":"https://arxiv.org/pdf/2401.12452v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"self-supervised-learning-of-lidar-3d-point","repo_url":"https://github.com/eaphan/nclr","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"self-supervised-learning-of-lidar-3d-point","repo_url":"https://github.com/rsy6318/CorrI2P","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"3d-semantic-segmentation","task_name":"3D Semantic Segmentation"},{"task_slug":"autonomous-driving","task_name":"Autonomous Driving"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"panoptic-segmentation","task_name":"Panoptic Segmentation"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.12452","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.12452"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rsy6318/CorrI2P","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/eaphan/nclr","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":17,"ran_draft_wrong":3,"ran_fixture":3,"ran_violates":1,"unverified":2},"by_repo_kind":{"official":{"samples":11,"ran":10,"repositories":1},"listed":{"samples":12,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":26,"samples":[{"code_sha256_prefix":"fc333893f7271b90","entry":"FPS","repo":"rsy6318/CorrI2P","repo_kind":"listed","path":"pointnet.py","file_url":"https://github.com/rsy6318/CorrI2P/blob/HEAD/pointnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fc333893f7271b90"}},{"code_sha256_prefix":"60f2f4207ae20ab9","entry":"angles2rotation_matrix","repo":"rsy6318/CorrI2P","repo_kind":"listed","path":"nuscenes_pc_img_dataloader.py","file_url":"https://github.com/rsy6318/CorrI2P/blob/HEAD/nuscenes_pc_img_dataloader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"60f2f4207ae20ab9"}},{"code_sha256_prefix":"d30c03f27e494c3c","entry":"bilinear_interpolate_torch","repo":"eaphan/nclr","repo_kind":"official","path":"bevlab/models.py","file_url":"https://github.com/eaphan/nclr/blob/HEAD/bevlab/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d30c03f27e494c3c"}},{"code_sha256_prefix":"80730c3bb0cf7118","entry":"camera_matrix_cropping","repo":"rsy6318/CorrI2P","repo_kind":"listed","path":"nuscenes_pc_img_dataloader.py","file_url":"https://github.com/rsy6318/CorrI2P/blob/HEAD/nuscenes_pc_img_dataloader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"80730c3bb0cf7118"}},{"code_sha256_prefix":"857b8f722d0dcda1","entry":"camera_matrix_scaling","repo":"rsy6318/CorrI2P","repo_kind":"listed","path":"nuscenes_pc_img_dataloader.py","file_url":"https://github.com/rsy6318/CorrI2P/blob/HEAD/nuscenes_pc_img_dataloader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"857b8f722d0dcda1"}},{"code_sha256_prefix":"d9def42110729a85","entry":"conv1x1","repo":"rsy6318/CorrI2P","repo_kind":"listed","path":"imagenet.py","file_url":"https://github.com/rsy6318/CorrI2P/blob/HEAD/imagenet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d9def42110729a85"}},{"code_sha256_prefix":"160bb14bd76201b4","entry":"conv3x3","repo":"rsy6318/CorrI2P","repo_kind":"listed","path":"imagenet.py","file_url":"https://github.com/rsy6318/CorrI2P/blob/HEAD/imagenet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"160bb14bd76201b4"}},{"code_sha256_prefix":"fd2ccb302a5c2310","entry":"convert_pointclouds_to_tensor","repo":"eaphan/nclr","repo_kind":"official","path":"bevlab/pnp.py","file_url":"https://github.com/eaphan/nclr/blob/HEAD/bevlab/pnp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"fd2ccb302a5c2310"}},{"code_sha256_prefix":"3d7f0e7dd8fdeb74","entry":"create_correspondence_matrix","repo":"eaphan/nclr","repo_kind":"official","path":"bevlab/models.py","file_url":"https://github.com/eaphan/nclr/blob/HEAD/bevlab/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3d7f0e7dd8fdeb74"}},{"code_sha256_prefix":"4af84e33621075ff","entry":"desc_loss2","repo":"rsy6318/CorrI2P","repo_kind":"listed","path":"loss.py","file_url":"https://github.com/rsy6318/CorrI2P/blob/HEAD/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4af84e33621075ff"}},{"code_sha256_prefix":"77bbbfa9a1d1b7f5","entry":"det_loss","repo":"rsy6318/CorrI2P","repo_kind":"listed","path":"loss.py","file_url":"https://github.com/rsy6318/CorrI2P/blob/HEAD/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"77bbbfa9a1d1b7f5"}},{"code_sha256_prefix":"408a3d02069cff3a","entry":"eigh","repo":"eaphan/nclr","repo_kind":"official","path":"bevlab/pnp.py","file_url":"https://github.com/eaphan/nclr/blob/HEAD/bevlab/pnp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"408a3d02069cff3a"}},{"code_sha256_prefix":"685179cc383d2145","entry":"get_P_diff","repo":"rsy6318/CorrI2P","repo_kind":"listed","path":"cal_error_all.py","file_url":"https://github.com/rsy6318/CorrI2P/blob/HEAD/cal_error_all.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"685179cc383d2145"}},{"code_sha256_prefix":"497ee1ee5d05dcd0","entry":"get_P_from_Rt","repo":"eaphan/nclr","repo_kind":"official","path":"bevlab/datasets.py","file_url":"https://github.com/eaphan/nclr/blob/HEAD/bevlab/datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"497ee1ee5d05dcd0"}},{"code_sha256_prefix":"6fef16c520722b39","entry":"get_savedir_name","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"6fef16c520722b39"}},{"code_sha256_prefix":"573cb024ea47999f","entry":"index_points","repo":"rsy6318/CorrI2P","repo_kind":"listed","path":"pointnet.py","file_url":"https://github.com/rsy6318/CorrI2P/blob/HEAD/pointnet.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"573cb024ea47999f"}},{"code_sha256_prefix":"70596f28650426a1","entry":"interpolate_from_bev_features","repo":"eaphan/nclr","repo_kind":"official","path":"bevlab/models.py","file_url":"https://github.com/eaphan/nclr/blob/HEAD/bevlab/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"70596f28650426a1"}},{"code_sha256_prefix":"d08e64b2d73c665b","entry":"is_pointclouds","repo":"eaphan/nclr","repo_kind":"official","path":"bevlab/pnp.py","file_url":"https://github.com/eaphan/nclr/blob/HEAD/bevlab/pnp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d08e64b2d73c665b"}},{"code_sha256_prefix":"5f4f4355be28c01e","entry":"isnan","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"5f4f4355be28c01e"}},{"code_sha256_prefix":"ed2a67a7b19a5c07","entry":"mean","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"ed2a67a7b19a5c07"}},{"code_sha256_prefix":"def5efb3f261d1c1","entry":"revtrans_jittering","repo":"eaphan/nclr","repo_kind":"official","path":"bevlab/transforms.py","file_url":"https://github.com/eaphan/nclr/blob/HEAD/bevlab/transforms.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"def5efb3f261d1c1"}},{"code_sha256_prefix":"ce2dbefef7a9a194","entry":"revtrans_rotation","repo":"eaphan/nclr","repo_kind":"official","path":"bevlab/transforms.py","file_url":"https://github.com/eaphan/nclr/blob/HEAD/bevlab/transforms.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ce2dbefef7a9a194"}},{"code_sha256_prefix":"d8fbeeb1bc0bfeee","entry":"revtrans_translation","repo":"eaphan/nclr","repo_kind":"official","path":"bevlab/transforms.py","file_url":"https://github.com/eaphan/nclr/blob/HEAD/bevlab/transforms.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d8fbeeb1bc0bfeee"}},{"code_sha256_prefix":"580dee6761649b29","entry":"square_distance","repo":"rsy6318/CorrI2P","repo_kind":"listed","path":"pointnet.py","file_url":"https://github.com/rsy6318/CorrI2P/blob/HEAD/pointnet.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"580dee6761649b29"}},{"code_sha256_prefix":"6f7320d8f113ebd6","entry":"adapt_weights","repo":"eaphan/nclr","repo_kind":"official","path":"bevlab/imagenet.py","file_url":"https://github.com/eaphan/nclr/blob/HEAD/bevlab/imagenet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6f7320d8f113ebd6"}},{"code_sha256_prefix":"457b688bce1b0ffe","entry":"desc_loss","repo":"rsy6318/CorrI2P","repo_kind":"listed","path":"loss.py","file_url":"https://github.com/rsy6318/CorrI2P/blob/HEAD/loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"457b688bce1b0ffe"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}