{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/geminifusion-efficient-pixel-wise-multimodal","title":"GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformer","arxiv_id":"2406.01210","date":"2024-06-03","proceeding":null,"authors":["Ding Jia","Jianyuan Guo","Kai Han","Han Wu","Chao Zhang","Chang Xu","Xinghao Chen"],"abstract":"Cross-modal transformers have demonstrated superiority in various vision tasks by effectively integrating different modalities. This paper first critiques prior token exchange methods which replace less informative tokens with inter-modal features, and demonstrate exchange based methods underperform cross-attention mechanisms, while the computational demand of the latter inevitably restricts its use with longer sequences. To surmount the computational challenges, we propose GeminiFusion, a pixel-wise fusion approach that capitalizes on aligned cross-modal representations. GeminiFusion elegantly combines intra-modal and inter-modal attentions, dynamically integrating complementary information across modalities. We employ a layer-adaptive noise to adaptively control their interplay on a per-layer basis, thereby achieving a harmonized fusion process. Notably, GeminiFusion maintains linear complexity with respect to the number of input tokens, ensuring this multimodal framework operates with efficiency comparable to unimodal networks. Comprehensive evaluations across multimodal image-to-image translation, 3D object detection and arbitrary-modal semantic segmentation tasks, including RGB, depth, LiDAR, event data, etc. demonstrate the superior performance of our GeminiFusion against leading-edge techniques. The PyTorch code is available at https://github.com/JiaDingCN/GeminiFusion","url_abs":"https://arxiv.org/abs/2406.01210v2","url_pdf":"https://arxiv.org/pdf/2406.01210v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"geminifusion-efficient-pixel-wise-multimodal","repo_url":"https://github.com/jiadingcn/geminifusion","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"3d-object-detection","task_name":"3D Object Detection"},{"task_slug":"image-to-image-translation","task_name":"Image-to-Image Translation"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/semantic-segmentation-on-deliver-1","task":"Semantic Segmentation","dataset":"DELIVER","model":"GeminiFusion","rank_in_archive_order":2,"of":9,"metrics":{"mIoU":"66.9"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-deliver","task":"Semantic Segmentation","dataset":"DeLiVER","model":"GeminiFusion","rank_in_archive_order":3,"of":26,"metrics":{"mIoU":"66.9"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-nyu-depth-v2","task":"Semantic Segmentation","dataset":"NYU Depth v2","model":"GeminiFusion (Swin-Large)","rank_in_archive_order":4,"of":121,"metrics":{"Mean IoU":"60.9"},"uses_additional_data":true},{"leaderboard":"/sota/semantic-segmentation-on-nyu-depth-v2","task":"Semantic Segmentation","dataset":"NYU Depth v2","model":"GeminiFusion (Swin-Large)","rank_in_archive_order":6,"of":121,"metrics":{"Mean IoU":"60.2"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-nyu-depth-v2","task":"Semantic Segmentation","dataset":"NYU Depth v2","model":"GeminiFusion (MiT-B5)","rank_in_archive_order":14,"of":121,"metrics":{"Mean IoU":"57.7"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-nyu-depth-v2","task":"Semantic Segmentation","dataset":"NYU Depth v2","model":"GeminiFusion (MiT-B3)","rank_in_archive_order":20,"of":121,"metrics":{"Mean IoU":"56.8"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-sun-rgbd","task":"Semantic Segmentation","dataset":"SUN-RGBD","model":"GeminiFusion (Swin-Large)","rank_in_archive_order":1,"of":44,"metrics":{"Mean IoU":"54.6"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-sun-rgbd","task":"Semantic Segmentation","dataset":"SUN-RGBD","model":"GeminiFusion (MiT-B5)","rank_in_archive_order":4,"of":44,"metrics":{"Mean IoU":"53.3"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-sun-rgbd","task":"Semantic Segmentation","dataset":"SUN-RGBD","model":"GeminiFusion (MiT-B3)","rank_in_archive_order":9,"of":44,"metrics":{"Mean IoU":"52.7"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.01210","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.01210"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jiadingcn/geminifusion","reach":null}],"summary":{"ran_fixture":1,"ran":5,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":10,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"144d10b49baeb8a6","entry":"window_partition","repo":"JiaDingCN/GeminiFusion","repo_kind":"official","path":"models/swin_transformer.py","file_url":"https://github.com/JiaDingCN/GeminiFusion/blob/HEAD/models/swin_transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"144d10b49baeb8a6"}},{"code_sha256_prefix":"b8aa506cdb9aa1f4","entry":"LayerNormParallel","repo":"jiadingcn/geminifusion","repo_kind":"official","path":"models/mix_transformer.py","file_url":"https://github.com/jiadingcn/geminifusion/blob/HEAD/models/mix_transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b8aa506cdb9aa1f4"}},{"code_sha256_prefix":"0cde8e101baaac37","entry":"Mlp","repo":"jiadingcn/geminifusion","repo_kind":"official","path":"models/mix_transformer.py","file_url":"https://github.com/jiadingcn/geminifusion/blob/HEAD/models/mix_transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0cde8e101baaac37"}},{"code_sha256_prefix":"6b642fcee761770c","entry":"Mlp_2","repo":"jiadingcn/geminifusion","repo_kind":"official","path":"models/mix_transformer.py","file_url":"https://github.com/jiadingcn/geminifusion/blob/HEAD/models/mix_transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6b642fcee761770c"}},{"code_sha256_prefix":"eea45a75f069aff9","entry":"ModuleParallel","repo":"jiadingcn/geminifusion","repo_kind":"official","path":"models/mix_transformer.py","file_url":"https://github.com/jiadingcn/geminifusion/blob/HEAD/models/mix_transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eea45a75f069aff9"}},{"code_sha256_prefix":"d9f8697f8d4a3b0f","entry":"OverlapPatchEmbed","repo":"jiadingcn/geminifusion","repo_kind":"official","path":"models/mix_transformer.py","file_url":"https://github.com/jiadingcn/geminifusion/blob/HEAD/models/mix_transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d9f8697f8d4a3b0f"}},{"code_sha256_prefix":"61bf152e6a42a184","entry":"window_reverse","repo":"JiaDingCN/GeminiFusion","repo_kind":"official","path":"models/swin_transformer.py","file_url":"https://github.com/JiaDingCN/GeminiFusion/blob/HEAD/models/swin_transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"61bf152e6a42a184"}},{"code_sha256_prefix":"19d28354dedaaeba","entry":"Attention","repo":"jiadingcn/geminifusion","repo_kind":"official","path":"models/mix_transformer.py","file_url":"https://github.com/jiadingcn/geminifusion/blob/HEAD/models/mix_transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"19d28354dedaaeba"}},{"code_sha256_prefix":"ddc545386a46bb98","entry":"Block","repo":"jiadingcn/geminifusion","repo_kind":"official","path":"models/mix_transformer.py","file_url":"https://github.com/jiadingcn/geminifusion/blob/HEAD/models/mix_transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ddc545386a46bb98"}},{"code_sha256_prefix":"25f4c1d795867ecb","entry":"MixVisionTransformer","repo":"jiadingcn/geminifusion","repo_kind":"official","path":"models/mix_transformer.py","file_url":"https://github.com/jiadingcn/geminifusion/blob/HEAD/models/mix_transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"25f4c1d795867ecb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}