{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multimodal-object-detection-via-bayesian","title":"Multimodal Object Detection via Probabilistic Ensembling","arxiv_id":"2104.02904","date":"2021-04-07","proceeding":null,"authors":["Yi-Ting Chen","Jinghao Shi","Zelin Ye","Christoph Mertz","Deva Ramanan","Shu Kong"],"abstract":"Object detection with multimodal inputs can improve many safety-critical systems such as autonomous vehicles (AVs). Motivated by AVs that operate in both day and night, we study multimodal object detection with RGB and thermal cameras, since the latter provides much stronger object signatures under poor illumination. We explore strategies for fusing information from different modalities. Our key contribution is a probabilistic ensembling technique, ProbEn, a simple non-learned method that fuses together detections from multi-modalities. We derive ProbEn from Bayes' rule and first principles that assume conditional independence across modalities. Through probabilistic marginalization, ProbEn elegantly handles missing modalities when detectors do not fire on the same object. Importantly, ProbEn also notably improves multimodal detection even when the conditional independence assumption does not hold, e.g., fusing outputs from other fusion methods (both off-the-shelf and trained in-house). We validate ProbEn on two benchmarks containing both aligned (KAIST) and unaligned (FLIR) multimodal images, showing that ProbEn outperforms prior work by more than 13% in relative performance!","url_abs":"https://arxiv.org/abs/2104.02904v3","url_pdf":"https://arxiv.org/pdf/2104.02904v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multimodal-object-detection-via-bayesian","repo_url":"https://github.com/Jamie725/RGBT-detection","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"multimodal-object-detection-via-bayesian","repo_url":"https://github.com/Jamie725/Multimodal-Object-Detection-via-Probabilistic-Ensembling","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"multimodal-object-detection-via-bayesian","repo_url":"https://github.com/aimerykong/RGBT-detection","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"3d-object-detection","task_name":"3D Object Detection"},{"task_slug":"autonomous-vehicles","task_name":"Autonomous Vehicles"},{"task_slug":"object","task_name":"Object"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"pedestrian-detection","task_name":"Pedestrian Detection"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/object-detection-on-eventped","task":"Object Detection","dataset":"EventPed","model":"ProEN","rank_in_archive_order":3,"of":6,"metrics":{"AP":"60.1"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-on-inoutdoor","task":"Object Detection","dataset":"InOutDoor","model":"ProbEN","rank_in_archive_order":2,"of":6,"metrics":{"AP":"62.4"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-on-stcrowd","task":"Object Detection","dataset":"STCrowd","model":"ProbEN","rank_in_archive_order":4,"of":6,"metrics":{"AP":"60.0"},"uses_additional_data":false},{"leaderboard":"/sota/pedestrian-detection-on-llvip","task":"Pedestrian Detection","dataset":"LLVIP","model":"ProbEn","rank_in_archive_order":13,"of":15,"metrics":{"AP":"0.515"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2104.02904","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.02904"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Jamie725/RGBT-detection","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Jamie725/Multimodal-Object-Detection-via-Probabilistic-Ensembling","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aimerykong/RGBT-detection","reach":{"status":"ok"}}],"summary":{"ran_honours":2,"ran":5,"ran_fixture":1,"unverified":3},"by_repo_kind":{"official":{"samples":8,"ran":5,"repositories":1},"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"37529abc553ac28b","entry":"avg_bbox_fusion","repo":"Jamie725/Multimodal-Object-Detection-via-Probabilistic-Ensembling","repo_kind":"listed","path":"demo/FLIR/demo_probEn.py","file_url":"https://github.com/Jamie725/Multimodal-Object-Detection-via-Probabilistic-Ensembling/blob/HEAD/demo/FLIR/demo_probEn.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"37529abc553ac28b"}},{"code_sha256_prefix":"0df9d3b583fbc6cb","entry":"batched_nms","repo":"Jamie725/RGBT-detection","repo_kind":"official","path":"detectron2/layers/nms.py","file_url":"https://github.com/Jamie725/RGBT-detection/blob/HEAD/detectron2/layers/nms.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"0df9d3b583fbc6cb"}},{"code_sha256_prefix":"e1b20ede1f3d84ad","entry":"bayesian_fusion","repo":"Jamie725/Multimodal-Object-Detection-via-Probabilistic-Ensembling","repo_kind":"listed","path":"demo/FLIR/demo_probEn.py","file_url":"https://github.com/Jamie725/Multimodal-Object-Detection-via-Probabilistic-Ensembling/blob/HEAD/demo/FLIR/demo_probEn.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e1b20ede1f3d84ad"}},{"code_sha256_prefix":"1bfe4666270f8778","entry":"bayesian_fusion_multiclass","repo":"Jamie725/Multimodal-Object-Detection-via-Probabilistic-Ensembling","repo_kind":"listed","path":"demo/FLIR/demo_probEn.py","file_url":"https://github.com/Jamie725/Multimodal-Object-Detection-via-Probabilistic-Ensembling/blob/HEAD/demo/FLIR/demo_probEn.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1bfe4666270f8778"}},{"code_sha256_prefix":"e8526a516b4f9206","entry":"convert_basic_c2_names","repo":"Jamie725/RGBT-detection","repo_kind":"official","path":"detectron2/checkpoint/c2_model_loading.py","file_url":"https://github.com/Jamie725/RGBT-detection/blob/HEAD/detectron2/checkpoint/c2_model_loading.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"e8526a516b4f9206"}},{"code_sha256_prefix":"52810b504f0d1066","entry":"convert_c2_detectron_names","repo":"Jamie725/RGBT-detection","repo_kind":"official","path":"detectron2/checkpoint/c2_model_loading.py","file_url":"https://github.com/Jamie725/RGBT-detection/blob/HEAD/detectron2/checkpoint/c2_model_loading.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"52810b504f0d1066"}},{"code_sha256_prefix":"406f84cf799ed34d","entry":"pad_masks","repo":"Jamie725/RGBT-detection","repo_kind":"official","path":"detectron2/layers/mask_ops.py","file_url":"https://github.com/Jamie725/RGBT-detection/blob/HEAD/detectron2/layers/mask_ops.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"406f84cf799ed34d"}},{"code_sha256_prefix":"d03592674030077f","entry":"paste_mask_in_image_old","repo":"Jamie725/RGBT-detection","repo_kind":"official","path":"detectron2/layers/mask_ops.py","file_url":"https://github.com/Jamie725/RGBT-detection/blob/HEAD/detectron2/layers/mask_ops.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"d03592674030077f"}},{"code_sha256_prefix":"4845964b76bba6c5","entry":"batched_nms_rotated","repo":"Jamie725/RGBT-detection","repo_kind":"official","path":"detectron2/layers/nms.py","file_url":"https://github.com/Jamie725/RGBT-detection/blob/HEAD/detectron2/layers/nms.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"4845964b76bba6c5"}},{"code_sha256_prefix":"d3fa37a000ad294d","entry":"nms_rotated","repo":"Jamie725/RGBT-detection","repo_kind":"official","path":"detectron2/layers/nms.py","file_url":"https://github.com/Jamie725/RGBT-detection/blob/HEAD/detectron2/layers/nms.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"d3fa37a000ad294d"}},{"code_sha256_prefix":"13b3f33534eea02a","entry":"paste_masks_in_image","repo":"Jamie725/RGBT-detection","repo_kind":"official","path":"detectron2/layers/mask_ops.py","file_url":"https://github.com/Jamie725/RGBT-detection/blob/HEAD/detectron2/layers/mask_ops.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"13b3f33534eea02a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}