{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/is-fusion-instance-scene-collaborative-fusion","title":"IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection","arxiv_id":"2403.15241","date":"2024-03-22","proceeding":"CVPR 2024 1","authors":["Junbo Yin","Jianbing Shen","Runnan Chen","Wei Li","Ruigang Yang","Pascal Frossard","Wenguan Wang"],"abstract":"Bird's eye view (BEV) representation has emerged as a dominant solution for describing 3D space in autonomous driving scenarios. However, objects in the BEV representation typically exhibit small sizes, and the associated point cloud context is inherently sparse, which leads to great challenges for reliable 3D perception. In this paper, we propose IS-Fusion, an innovative multimodal fusion framework that jointly captures the Instance- and Scene-level contextual information. IS-Fusion essentially differs from existing approaches that only focus on the BEV scene-level fusion by explicitly incorporating instance-level multimodal information, thus facilitating the instance-centric tasks like 3D object detection. It comprises a Hierarchical Scene Fusion (HSF) module and an Instance-Guided Fusion (IGF) module. HSF applies Point-to-Grid and Grid-to-Region transformers to capture the multimodal scene context at different granularities. IGF mines instance candidates, explores their relationships, and aggregates the local multimodal context for each instance. These instances then serve as guidance to enhance the scene feature and yield an instance-aware BEV representation. On the challenging nuScenes benchmark, IS-Fusion outperforms all the published multimodal works to date. Code is available at: https://github.com/yinjunbo/IS-Fusion.","url_abs":"https://arxiv.org/abs/2403.15241v1","url_pdf":"https://arxiv.org/pdf/2403.15241v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"is-fusion-instance-scene-collaborative-fusion","repo_url":"https://github.com/yinjunbo/is-fusion","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"3d-object-detection","task_name":"3D Object Detection"},{"task_slug":"autonomous-driving","task_name":"Autonomous Driving"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.15241","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.15241"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yinjunbo/is-fusion","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/yinjunbo/IS-Fusion","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3,"ran_fixture":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3fda7e126777fae8","entry":"bbox_2d_transform","repo":"yinjunbo/IS-Fusion","repo_kind":"official","path":"mmdet3d/models/fusion_layers/coord_transform.py","file_url":"https://github.com/yinjunbo/IS-Fusion/blob/HEAD/mmdet3d/models/fusion_layers/coord_transform.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3fda7e126777fae8"}},{"code_sha256_prefix":"5a5d8bac063f49f1","entry":"extract_2d_info","repo":"yinjunbo/IS-Fusion","repo_kind":"official","path":"mmdet3d/models/fusion_layers/coord_transform.py","file_url":"https://github.com/yinjunbo/IS-Fusion/blob/HEAD/mmdet3d/models/fusion_layers/coord_transform.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5a5d8bac063f49f1"}},{"code_sha256_prefix":"4c82654ae6131ecc","entry":"gen_sineembed_for_position","repo":"yinjunbo/IS-Fusion","repo_kind":"official","path":"mmdet3d/models/middle_encoders/fusion_encoder.py","file_url":"https://github.com/yinjunbo/IS-Fusion/blob/HEAD/mmdet3d/models/middle_encoders/fusion_encoder.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4c82654ae6131ecc"}},{"code_sha256_prefix":"e8974f996d0fa2ae","entry":"multi_head_attention_forward","repo":"yinjunbo/IS-Fusion","repo_kind":"official","path":"mmdet3d/models/middle_encoders/fusion_encoder.py","file_url":"https://github.com/yinjunbo/IS-Fusion/blob/HEAD/mmdet3d/models/middle_encoders/fusion_encoder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e8974f996d0fa2ae"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}