{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/advancing-self-supervised-monocular-depth","title":"Advancing Self-supervised Monocular Depth Learning with Sparse LiDAR","arxiv_id":"2109.09628","date":"2021-09-20","proceeding":null,"authors":["Ziyue Feng","Longlong Jing","Peng Yin","YingLi Tian","Bing Li"],"abstract":"Self-supervised monocular depth prediction provides a cost-effective solution to obtain the 3D location of each pixel. However, the existing approaches usually lead to unsatisfactory accuracy, which is critical for autonomous robots. In this paper, we propose FusionDepth, a novel two-stage network to advance the self-supervised monocular dense depth learning by leveraging low-cost sparse (e.g. 4-beam) LiDAR. Unlike the existing methods that use sparse LiDAR mainly in a manner of time-consuming iterative post-processing, our model fuses monocular image features and sparse LiDAR features to predict initial depth maps. Then, an efficient feed-forward refine network is further designed to correct the errors in these initial depth maps in pseudo-3D space with real-time performance. Extensive experiments show that our proposed model significantly outperforms all the state-of-the-art self-supervised methods, as well as the sparse-LiDAR-based methods on both self-supervised monocular depth prediction and completion tasks. With the accurate dense depth prediction, our model outperforms the state-of-the-art sparse-LiDAR-based method (Pseudo-LiDAR++) by more than 68% for the downstream task monocular 3D object detection on the KITTI Leaderboard. Code is available at https://github.com/AutoAILab/FusionDepth","url_abs":"https://arxiv.org/abs/2109.09628v4","url_pdf":"https://arxiv.org/pdf/2109.09628v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"advancing-self-supervised-monocular-depth","repo_url":"https://github.com/AutoAILab/FusionDepth","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"advancing-self-supervised-monocular-depth","repo_url":"https://github.com/fengziyue/FusionDepth","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"3d-object-detection","task_name":"3D Object Detection"},{"task_slug":"depth-completion","task_name":"Depth Completion"},{"task_slug":"depth-estimation","task_name":"Depth Estimation"},{"task_slug":"depth-prediction","task_name":"Depth Prediction"},{"task_slug":"monocular-3d-object-detection","task_name":"Monocular 3D Object Detection"},{"task_slug":"monocular-depth-estimation","task_name":"Monocular Depth Estimation"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/depth-completion-on-kitti","task":"Depth Completion","dataset":"KITTI","model":"FusionDepth","rank_in_archive_order":1,"of":1,"metrics":{"RMSE":"1193.92"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2109.09628","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2109.09628"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AutoAILab/FusionDepth","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fengziyue/FusionDepth","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"ran_honours":1,"ran":2,"unverified":11},"by_repo_kind":{"official":{"samples":15,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a8814cc5c348b987","entry":"batch_post_process_disparity","repo":"AutoAILab/FusionDepth","repo_kind":"official","path":"evaluate_completion.py","file_url":"https://github.com/AutoAILab/FusionDepth/blob/HEAD/evaluate_completion.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a8814cc5c348b987"}},{"code_sha256_prefix":"edd4f86e8f02732f","entry":"compute_errors","repo":"AutoAILab/FusionDepth","repo_kind":"official","path":"evaluate_depth.py","file_url":"https://github.com/AutoAILab/FusionDepth/blob/HEAD/evaluate_depth.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"edd4f86e8f02732f"}},{"code_sha256_prefix":"62287188376f0ba0","entry":"disp_to_depth","repo":"AutoAILab/FusionDepth","repo_kind":"official","path":"layers.py","file_url":"https://github.com/AutoAILab/FusionDepth/blob/HEAD/layers.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"62287188376f0ba0"}},{"code_sha256_prefix":"955112f5788539a8","entry":"get_translation_matrix","repo":"AutoAILab/FusionDepth","repo_kind":"official","path":"layers.py","file_url":"https://github.com/AutoAILab/FusionDepth/blob/HEAD/layers.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"955112f5788539a8"}},{"code_sha256_prefix":"84e9764cfc1ac6b1","entry":"bottom_crop","repo":"AutoAILab/FusionDepth","repo_kind":"official","path":"gen2cha_completion.py","file_url":"https://github.com/AutoAILab/FusionDepth/blob/HEAD/gen2cha_completion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"84e9764cfc1ac6b1"}},{"code_sha256_prefix":"8e520332ac6e2b25","entry":"compute_errors","repo":"AutoAILab/FusionDepth","repo_kind":"official","path":"evaluate_completion.py","file_url":"https://github.com/AutoAILab/FusionDepth/blob/HEAD/evaluate_completion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8e520332ac6e2b25"}},{"code_sha256_prefix":"726548552574d958","entry":"depth2ptc","repo":"AutoAILab/FusionDepth","repo_kind":"official","path":"evaluate_depth.py","file_url":"https://github.com/AutoAILab/FusionDepth/blob/HEAD/evaluate_depth.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"726548552574d958"}},{"code_sha256_prefix":"e1d7378c4e2e8fce","entry":"filter_mask","repo":"AutoAILab/FusionDepth","repo_kind":"official","path":"gdc_old.py","file_url":"https://github.com/AutoAILab/FusionDepth/blob/HEAD/gdc_old.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e1d7378c4e2e8fce"}},{"code_sha256_prefix":"c2ce62aec738dd30","entry":"filter_theta_mask","repo":"AutoAILab/FusionDepth","repo_kind":"official","path":"gdc_old.py","file_url":"https://github.com/AutoAILab/FusionDepth/blob/HEAD/gdc_old.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c2ce62aec738dd30"}},{"code_sha256_prefix":"7020a51528198a64","entry":"get_4beam_2channel","repo":"AutoAILab/FusionDepth","repo_kind":"official","path":"gen2cha_completion.py","file_url":"https://github.com/AutoAILab/FusionDepth/blob/HEAD/gen2cha_completion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7020a51528198a64"}},{"code_sha256_prefix":"7069e34106de8ca2","entry":"get_4beam_2channel","repo":"AutoAILab/FusionDepth","repo_kind":"official","path":"gen2channel.py","file_url":"https://github.com/AutoAILab/FusionDepth/blob/HEAD/gen2channel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7069e34106de8ca2"}},{"code_sha256_prefix":"13035b58e28992f7","entry":"get_depth","repo":"AutoAILab/FusionDepth","repo_kind":"official","path":"gen2cha_completion.py","file_url":"https://github.com/AutoAILab/FusionDepth/blob/HEAD/gen2cha_completion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"13035b58e28992f7"}},{"code_sha256_prefix":"57ea01679c8b00e5","entry":"get_detec_calib","repo":"AutoAILab/FusionDepth","repo_kind":"official","path":"export_detection.py","file_url":"https://github.com/AutoAILab/FusionDepth/blob/HEAD/export_detection.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"57ea01679c8b00e5"}},{"code_sha256_prefix":"0bb6694313ca5db7","entry":"subsample_mask_by_grid","repo":"AutoAILab/FusionDepth","repo_kind":"official","path":"gdc_old.py","file_url":"https://github.com/AutoAILab/FusionDepth/blob/HEAD/gdc_old.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0bb6694313ca5db7"}},{"code_sha256_prefix":"03d0f03d4f4d3260","entry":"transformation_from_parameters","repo":"AutoAILab/FusionDepth","repo_kind":"official","path":"layers.py","file_url":"https://github.com/AutoAILab/FusionDepth/blob/HEAD/layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"03d0f03d4f4d3260"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}