{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/long-tailed-3d-detection-via-2d-late-fusion","title":"Long-Tailed 3D Detection via Multi-Modal Fusion","arxiv_id":"2312.10986","date":"2023-12-18","proceeding":null,"authors":["Yechi Ma","Neehar Peri","Shuoquan Wei","Achal Dave","Wei Hua","Yanan Li","Deva Ramanan","Shu Kong"],"abstract":"Contemporary autonomous vehicle (AV) benchmarks have advanced techniques for training 3D detectors, particularly on large-scale multi-modal (LiDAR + RGB) data. Surprisingly, although semantic class labels naturally follow a long-tailed distribution, existing benchmarks only focus on a few common classes (e.g., pedestrian and car) and neglect many rare but crucial classes (e.g., emergency vehicle and stroller). However, AVs must reliably detect both common and rare classes for safe operation in the open world. We address this challenge by formally studying the problem of Long-Tailed 3D Detection (LT3D), which evaluates all annotated classes, including those in-the-tail. We address LT3D with hierarchical losses that promote feature sharing across classes, and introduce diagnostic metrics that award partial credit to ``reasonable'' mistakes with respect to the semantic hierarchy (e.g., mistaking a child for an adult). Further, we point out that rare-class accuracy is particularly improved via multi-modal late fusion (MMLF) of independently trained uni-modal LiDAR and RGB detectors. Importantly, such an MMLF framework allows us to leverage large-scale uni-modal datasets (with more examples for rare classes) to train better uni-modal detectors, unlike prevailing end-to-end trained multi-modal detectors that require paired multi-modal data. Finally, we examine three critical components of our simple MMLF approach from first principles and investigate whether to train 2D or 3D RGB detectors for fusion, whether to match RGB and LiDAR detections in 3D or the projected 2D image plane, and how to fuse matched detections. Our proposed MMLF approach significantly improves LT3D performance over prior work, particularly improving rare class performance from 12.8 to 20.0 mAP!","url_abs":"https://arxiv.org/abs/2312.10986v4","url_pdf":"https://arxiv.org/pdf/2312.10986v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"long-tailed-3d-detection-via-2d-late-fusion","repo_url":"https://github.com/mayechi/lt3d-lf","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"3d-object-detection","task_name":"3D Object Detection"},{"task_slug":"autonomous-vehicles","task_name":"Autonomous Vehicles"},{"task_slug":"depth-estimation","task_name":"Depth Estimation"},{"task_slug":"diagnostic","task_name":"Diagnostic"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2312.10986","atlas_url":"https://app.syntology.ai/?focus=2312.10986","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.10986"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mayechi/lt3d-lf","reach":{"status":"ok"}}],"summary":{"ran":14,"unverified":1},"by_repo_kind":{"official":{"samples":15,"ran":14,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"480505311f7aaa80","entry":"cache_lookup","repo":"mayechi/lt3d-lf","repo_kind":"official","path":"analysis/stratified_lca0_ap.py","file_url":"https://github.com/mayechi/lt3d-lf/blob/HEAD/analysis/stratified_lca0_ap.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"480505311f7aaa80"}},{"code_sha256_prefix":"329acd6c9aaf5155","entry":"calibration_from_json","repo":"mayechi/lt3d-lf","repo_kind":"official","path":"lt3d_lf/calibration.py","file_url":"https://github.com/mayechi/lt3d-lf/blob/HEAD/lt3d_lf/calibration.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"329acd6c9aaf5155"}},{"code_sha256_prefix":"00236c9e7be34d8a","entry":"check_point_in_img","repo":"mayechi/lt3d-lf","repo_kind":"official","path":"lt3d_lf/fusion.py","file_url":"https://github.com/mayechi/lt3d-lf/blob/HEAD/lt3d_lf/fusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"00236c9e7be34d8a"}},{"code_sha256_prefix":"727b61f5c14e92dc","entry":"ego_distance","repo":"mayechi/lt3d-lf","repo_kind":"official","path":"analysis/stratified_lca0_ap.py","file_url":"https://github.com/mayechi/lt3d-lf/blob/HEAD/analysis/stratified_lca0_ap.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"727b61f5c14e92dc"}},{"code_sha256_prefix":"1e8adc563b758332","entry":"lca0","repo":"mayechi/lt3d-lf","repo_kind":"official","path":"lt3d_lf/metrics.py","file_url":"https://github.com/mayechi/lt3d-lf/blob/HEAD/lt3d_lf/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1e8adc563b758332"}},{"code_sha256_prefix":"357751b9f8e9f8a1","entry":"lidar2img","repo":"mayechi/lt3d-lf","repo_kind":"official","path":"lt3d_lf/fusion.py","file_url":"https://github.com/mayechi/lt3d-lf/blob/HEAD/lt3d_lf/fusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"357751b9f8e9f8a1"}},{"code_sha256_prefix":"15a57e02b1ee3c6c","entry":"lidar_count_for_gt","repo":"mayechi/lt3d-lf","repo_kind":"official","path":"analysis/stratified_lca0_ap.py","file_url":"https://github.com/mayechi/lt3d-lf/blob/HEAD/analysis/stratified_lca0_ap.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"15a57e02b1ee3c6c"}},{"code_sha256_prefix":"fe833b27ab49ae23","entry":"load","repo":"mayechi/lt3d-lf","repo_kind":"official","path":"lt3d_lf/utils.py","file_url":"https://github.com/mayechi/lt3d-lf/blob/HEAD/lt3d_lf/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe833b27ab49ae23"}},{"code_sha256_prefix":"8938b79a9f23a7bb","entry":"manifest_path","repo":"mayechi/lt3d-lf","repo_kind":"official","path":"lt3d_lf/directional_tune.py","file_url":"https://github.com/mayechi/lt3d-lf/blob/HEAD/lt3d_lf/directional_tune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8938b79a9f23a7bb"}},{"code_sha256_prefix":"817a354ced0f660f","entry":"progressbar","repo":"mayechi/lt3d-lf","repo_kind":"official","path":"lt3d_lf/utils.py","file_url":"https://github.com/mayechi/lt3d-lf/blob/HEAD/lt3d_lf/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"817a354ced0f660f"}},{"code_sha256_prefix":"773288d87dbf2ec3","entry":"rounded","repo":"mayechi/lt3d-lf","repo_kind":"official","path":"lt3d_lf/directional_tune.py","file_url":"https://github.com/mayechi/lt3d-lf/blob/HEAD/lt3d_lf/directional_tune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"773288d87dbf2ec3"}},{"code_sha256_prefix":"904dbf0f59c378c4","entry":"runtime_args","repo":"mayechi/lt3d-lf","repo_kind":"official","path":"lt3d_lf/tune.py","file_url":"https://github.com/mayechi/lt3d-lf/blob/HEAD/lt3d_lf/tune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"904dbf0f59c378c4"}},{"code_sha256_prefix":"42c80eb17c846d36","entry":"tag","repo":"mayechi/lt3d-lf","repo_kind":"official","path":"lt3d_lf/directional_tune.py","file_url":"https://github.com/mayechi/lt3d-lf/blob/HEAD/lt3d_lf/directional_tune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"42c80eb17c846d36"}},{"code_sha256_prefix":"41b153582ebe8c90","entry":"uniform_calibration","repo":"mayechi/lt3d-lf","repo_kind":"official","path":"lt3d_lf/tune.py","file_url":"https://github.com/mayechi/lt3d-lf/blob/HEAD/lt3d_lf/tune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"41b153582ebe8c90"}},{"code_sha256_prefix":"52c5507d10c79e35","entry":"unpack_predictions","repo":"mayechi/lt3d-lf","repo_kind":"official","path":"lt3d_lf/utils.py","file_url":"https://github.com/mayechi/lt3d-lf/blob/HEAD/lt3d_lf/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"52c5507d10c79e35"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}