{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/omnifusion-360-monocular-depth-estimation-via","title":"OmniFusion: 360 Monocular Depth Estimation via Geometry-Aware Fusion","arxiv_id":"2203.00838","date":"2022-03-02","proceeding":"CVPR 2022 1","authors":["Yuyan Li","Yuliang Guo","Zhixin Yan","Xinyu Huang","Ye Duan","Liu Ren"],"abstract":"A well-known challenge in applying deep-learning methods to omnidirectional images is spherical distortion. In dense regression tasks such as depth estimation, where structural details are required, using a vanilla CNN layer on the distorted 360 image results in undesired information loss. In this paper, we propose a 360 monocular depth estimation pipeline, OmniFusion, to tackle the spherical distortion issue. Our pipeline transforms a 360 image into less-distorted perspective patches (i.e. tangent images) to obtain patch-wise predictions via CNN, and then merge the patch-wise results for final output. To handle the discrepancy between patch-wise predictions which is a major issue affecting the merging quality, we propose a new framework with the following key components. First, we propose a geometry-aware feature fusion mechanism that combines 3D geometric features with 2D image features to compensate for the patch-wise discrepancy. Second, we employ the self-attention-based transformer architecture to conduct a global aggregation of patch-wise information, which further improves the consistency. Last, we introduce an iterative depth refinement mechanism, to further refine the estimated depth based on the more accurate geometric features. Experiments show that our method greatly mitigates the distortion issue, and achieves state-of-the-art performances on several 360 monocular depth estimation benchmark datasets.","url_abs":"https://arxiv.org/abs/2203.00838v2","url_pdf":"https://arxiv.org/pdf/2203.00838v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"omnifusion-360-monocular-depth-estimation-via","repo_url":"https://github.com/yuyanli0831/omnifusion","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"depth-estimation","task_name":"Depth Estimation"},{"task_slug":"monocular-depth-estimation","task_name":"Monocular Depth Estimation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/depth-estimation-on-stanford2d3d-panoramic","task":"Depth Estimation","dataset":"Stanford2D3D Panoramic","model":"OmniFusion (2-iter)","rank_in_archive_order":6,"of":18,"metrics":{"RMSE":"0.3474","absolute relative error":"0.095"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2203.00838","atlas_url":"https://app.syntology.ai/?focus=2203.00838","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.00838"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yuyanli0831/omnifusion","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/yuyanli0831/OmniFusion","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"ran_draft_wrong":2,"ran_fixture":1,"unverified":2},"by_repo_kind":{"official":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"07a6f99e2ae6aebd","entry":"ConvBnReLU_v2","repo":"yuyanli0831/OmniFusion","repo_kind":"official","path":"model/spherical_model.py","file_url":"https://github.com/yuyanli0831/OmniFusion/blob/HEAD/model/spherical_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"07a6f99e2ae6aebd"}},{"code_sha256_prefix":"283401be0f593a0b","entry":"Transformer_Block","repo":"yuyanli0831/OmniFusion","repo_kind":"official","path":"model/spherical_model.py","file_url":"https://github.com/yuyanli0831/OmniFusion/blob/HEAD/model/spherical_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"283401be0f593a0b"}},{"code_sha256_prefix":"20d6f3fd0f18a06f","entry":"Transformer_cascade","repo":"yuyanli0831/OmniFusion","repo_kind":"official","path":"model/spherical_model.py","file_url":"https://github.com/yuyanli0831/OmniFusion/blob/HEAD/model/spherical_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"20d6f3fd0f18a06f"}},{"code_sha256_prefix":"7b76d5b375a1572d","entry":"convert_bn","repo":"yuyanli0831/OmniFusion","repo_kind":"official","path":"model/spherical_model.py","file_url":"https://github.com/yuyanli0831/OmniFusion/blob/HEAD/model/spherical_model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b76d5b375a1572d"}},{"code_sha256_prefix":"6b2319b03a88bd68","entry":"convert_conv","repo":"yuyanli0831/OmniFusion","repo_kind":"official","path":"model/spherical_model.py","file_url":"https://github.com/yuyanli0831/OmniFusion/blob/HEAD/model/spherical_model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6b2319b03a88bd68"}},{"code_sha256_prefix":"1ed0c1d98add8eb0","entry":"equi2pers","repo":"yuyanli0831/OmniFusion","repo_kind":"official","path":"model/spherical_model.py","file_url":"https://github.com/yuyanli0831/OmniFusion/blob/HEAD/model/spherical_model.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1ed0c1d98add8eb0"}},{"code_sha256_prefix":"9125a8f96f43fae6","entry":"pers2equi","repo":"yuyanli0831/OmniFusion","repo_kind":"official","path":"model/spherical_model.py","file_url":"https://github.com/yuyanli0831/OmniFusion/blob/HEAD/model/spherical_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9125a8f96f43fae6"}},{"code_sha256_prefix":"f1fa6fed821b6473","entry":"spherical_fusion","repo":"yuyanli0831/OmniFusion","repo_kind":"official","path":"model/spherical_model.py","file_url":"https://github.com/yuyanli0831/OmniFusion/blob/HEAD/model/spherical_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1fa6fed821b6473"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}