{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/alignmif-geometry-aligned-multimodal-implicit","title":"AlignMiF: Geometry-Aligned Multimodal Implicit Field for LiDAR-Camera Joint Synthesis","arxiv_id":"2402.17483","date":"2024-02-27","proceeding":"CVPR 2024 1","authors":["Tao Tang","Guangrun Wang","Yixing Lao","Peng Chen","Jie Liu","Liang Lin","Kaicheng Yu","Xiaodan Liang"],"abstract":"Neural implicit fields have been a de facto standard in novel view synthesis. Recently, there exist some methods exploring fusing multiple modalities within a single field, aiming to share implicit features from different modalities to enhance reconstruction performance. However, these modalities often exhibit misaligned behaviors: optimizing for one modality, such as LiDAR, can adversely affect another, like camera performance, and vice versa. In this work, we conduct comprehensive analyses on the multimodal implicit field of LiDAR-camera joint synthesis, revealing the underlying issue lies in the misalignment of different sensors. Furthermore, we introduce AlignMiF, a geometrically aligned multimodal implicit field with two proposed modules: Geometry-Aware Alignment (GAA) and Shared Geometry Initialization (SGI). These modules effectively align the coarse geometry across different modalities, significantly enhancing the fusion process between LiDAR and camera data. Through extensive experiments across various datasets and scenes, we demonstrate the effectiveness of our approach in facilitating better interaction between LiDAR and camera modalities within a unified neural field. Specifically, our proposed AlignMiF, achieves remarkable improvement over recent implicit fusion methods (+2.01 and +3.11 image PSNR on the KITTI-360 and Waymo datasets) and consistently surpasses single modality performance (13.8% and 14.2% reduction in LiDAR Chamfer Distance on the respective datasets).","url_abs":"https://arxiv.org/abs/2402.17483v1","url_pdf":"https://arxiv.org/pdf/2402.17483v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"alignmif-geometry-aligned-multimodal-implicit","repo_url":"https://github.com/tangtaogo/alignmif","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"novel-view-synthesis","task_name":"Novel View Synthesis"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.17483","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.17483"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/tangtaogo/alignmif","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":8,"unverified":2},"by_repo_kind":{"official":{"samples":10,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c1cb6e3e76067a5d","entry":"dice_coeff","repo":"tangtaogo/alignmif","repo_kind":"official","path":"lidarnvs/unet.py","file_url":"https://github.com/tangtaogo/alignmif/blob/HEAD/lidarnvs/unet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c1cb6e3e76067a5d"}},{"code_sha256_prefix":"02b5df44eb5c21a3","entry":"dice_loss","repo":"tangtaogo/alignmif","repo_kind":"official","path":"lidarnvs/unet.py","file_url":"https://github.com/tangtaogo/alignmif/blob/HEAD/lidarnvs/unet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"02b5df44eb5c21a3"}},{"code_sha256_prefix":"cfac707d99413ea6","entry":"find_closest_label","repo":"tangtaogo/alignmif","repo_kind":"official","path":"lidarnerf/convert.py","file_url":"https://github.com/tangtaogo/alignmif/blob/HEAD/lidarnerf/convert.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cfac707d99413ea6"}},{"code_sha256_prefix":"1929a5b8c241cd48","entry":"fscore","repo":"tangtaogo/alignmif","repo_kind":"official","path":"extern/fscore.py","file_url":"https://github.com/tangtaogo/alignmif/blob/HEAD/extern/fscore.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1929a5b8c241cd48"}},{"code_sha256_prefix":"00979fadb02bf8df","entry":"huber_loss","repo":"tangtaogo/alignmif","repo_kind":"official","path":"lidarnerf/loss.py","file_url":"https://github.com/tangtaogo/alignmif/blob/HEAD/lidarnerf/loss.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"00979fadb02bf8df"}},{"code_sha256_prefix":"6657e863031cc206","entry":"lidar_to_pano_with_intensities","repo":"tangtaogo/alignmif","repo_kind":"official","path":"lidarnerf/convert.py","file_url":"https://github.com/tangtaogo/alignmif/blob/HEAD/lidarnerf/convert.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6657e863031cc206"}},{"code_sha256_prefix":"ddcd1a350d11a260","entry":"mape_loss","repo":"tangtaogo/alignmif","repo_kind":"official","path":"lidarnerf/loss.py","file_url":"https://github.com/tangtaogo/alignmif/blob/HEAD/lidarnerf/loss.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ddcd1a350d11a260"}},{"code_sha256_prefix":"9445687b9106e2f8","entry":"multiclass_dice_coeff","repo":"tangtaogo/alignmif","repo_kind":"official","path":"lidarnvs/unet.py","file_url":"https://github.com/tangtaogo/alignmif/blob/HEAD/lidarnvs/unet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9445687b9106e2f8"}},{"code_sha256_prefix":"0eaf72cf098311a6","entry":"get_encoder","repo":"tangtaogo/alignmif","repo_kind":"official","path":"lidarnerf/encoding.py","file_url":"https://github.com/tangtaogo/alignmif/blob/HEAD/lidarnerf/encoding.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0eaf72cf098311a6"}},{"code_sha256_prefix":"fcd40c17509b87bd","entry":"lidar_to_pano_with_intensities_with_bbox_mask","repo":"tangtaogo/alignmif","repo_kind":"official","path":"lidarnerf/convert.py","file_url":"https://github.com/tangtaogo/alignmif/blob/HEAD/lidarnerf/convert.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fcd40c17509b87bd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}