{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unleash-the-potential-of-image-branch-for-1","title":"Unleash the Potential of Image Branch for Cross-modal 3D Object Detection","arxiv_id":"2301.09077","date":"2023-01-22","proceeding":"NeurIPS 2023 11","authors":["Yifan Zhang","Qijian Zhang","Junhui Hou","Yixuan Yuan","Guoliang Xing"],"abstract":"To achieve reliable and precise scene understanding, autonomous vehicles typically incorporate multiple sensing modalities to capitalize on their complementary attributes. However, existing cross-modal 3D detectors do not fully utilize the image domain information to address the bottleneck issues of the LiDAR-based detectors. This paper presents a new cross-modal 3D object detector, namely UPIDet, which aims to unleash the potential of the image branch from two aspects. First, UPIDet introduces a new 2D auxiliary task called normalized local coordinate map estimation. This approach enables the learning of local spatial-aware features from the image modality to supplement sparse point clouds. Second, we discover that the representational capability of the point cloud backbone can be enhanced through the gradients backpropagated from the training objectives of the image branch, utilizing a succinct and effective point-to-pixel module. Extensive experiments and ablation studies validate the effectiveness of our method. Notably, we achieved the top rank in the highly competitive cyclist class of the KITTI benchmark at the time of submission. The source code is available at https://github.com/Eaphan/UPIDet.","url_abs":"https://arxiv.org/abs/2301.09077v3","url_pdf":"https://arxiv.org/pdf/2301.09077v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unleash-the-potential-of-image-branch-for-1","repo_url":"https://github.com/eaphan/upidet","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"3d-object-detection","task_name":"3D Object Detection"},{"task_slug":"autonomous-vehicles","task_name":"Autonomous Vehicles"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"scene-understanding","task_name":"Scene Understanding"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2301.09077","atlas_url":"https://app.syntology.ai/?focus=2301.09077","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2301.09077"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Eaphan/UPIDet","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/eaphan/upidet","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"unverified":1},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6eee488b20bd778e","entry":"VFETemplate","repo":"eaphan/upidet","repo_kind":"official","path":"pcdet/models/backbones_3d/vfe/image_vfe.py","file_url":"https://github.com/eaphan/upidet/blob/HEAD/pcdet/models/backbones_3d/vfe/image_vfe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6eee488b20bd778e"}},{"code_sha256_prefix":"e03872d04c69c06d","entry":"ImageVFE","repo":"eaphan/upidet","repo_kind":"official","path":"pcdet/models/backbones_3d/vfe/image_vfe.py","file_url":"https://github.com/eaphan/upidet/blob/HEAD/pcdet/models/backbones_3d/vfe/image_vfe.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e03872d04c69c06d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}