{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mmtm-multimodal-transfer-module-for-cnn","title":"MMTM: Multimodal Transfer Module for CNN Fusion","arxiv_id":"1911.08670","date":"2019-11-20","proceeding":"CVPR 2020 6","authors":["Hamid Reza Vaezi Joze","Amirreza Shaban","Michael L. Iuzzolino","Kazuhito Koishida"],"abstract":"In late fusion, each modality is processed in a separate unimodal Convolutional Neural Network (CNN) stream and the scores of each modality are fused at the end. Due to its simplicity late fusion is still the predominant approach in many state-of-the-art multimodal applications. In this paper, we present a simple neural network module for leveraging the knowledge from multiple modalities in convolutional neural networks. The propose unit, named Multimodal Transfer Module (MMTM), can be added at different levels of the feature hierarchy, enabling slow modality fusion. Using squeeze and excitation operations, MMTM utilizes the knowledge of multiple modalities to recalibrate the channel-wise features in each CNN stream. Despite other intermediate fusion methods, the proposed module could be used for feature modality fusion in convolution layers with different spatial dimensions. Another advantage of the proposed method is that it could be added among unimodal branches with minimum changes in the their network architectures, allowing each branch to be initialized with existing pretrained weights. Experimental results show that our framework improves the recognition accuracy of well-known multimodal networks. We demonstrate state-of-the-art or competitive performance on four datasets that span the task domains of dynamic hand gesture recognition, speech enhancement, and action recognition with RGB and body joints.","url_abs":"https://arxiv.org/abs/1911.08670v2","url_pdf":"https://arxiv.org/pdf/1911.08670v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mmtm-multimodal-transfer-module-for-cnn","repo_url":"https://github.com/haamoon/mmtm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"action-recognition-in-videos-2","task_name":"Action Recognition In Videos"},{"task_slug":"gesture-recognition","task_name":"Gesture Recognition"},{"task_slug":"hand-gesture-recognition","task_name":"Hand Gesture Recognition"},{"task_slug":"hand-gesture-recognition-1","task_name":"Hand-Gesture Recognition"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"speech-enhancement","task_name":"Speech Enhancement"}],"methods":[{"method_slug":"convolution","method_name":"Convolution"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-recognition-in-videos-on-ntu-rgbd","task":"Action Recognition","dataset":"NTU RGB+D","model":"MMTM (RGB+Pose)","rank_in_archive_order":21,"of":28,"metrics":{"Accuracy (CS)":"91.99"},"uses_additional_data":false},{"leaderboard":"/sota/hand-gesture-recognition-on-egogesture-1","task":"Hand Gesture Recognition","dataset":"EgoGesture","model":"MMTM","rank_in_archive_order":3,"of":4,"metrics":{"Accuracy":"93.51"},"uses_additional_data":false},{"leaderboard":"/sota/hand-gesture-recognition-on-nvgesture-1","task":"Hand Gesture Recognition","dataset":"NVGesture","model":"MMTM","rank_in_archive_order":4,"of":5,"metrics":{"Accuracy":"86.93"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1911.08670","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1911.08670"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/haamoon/mmtm","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":3},"by_repo_kind":{"listed":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f5cbdca0600a0fd4","entry":"step","repo":"haamoon/mmtm","repo_kind":"listed","path":"main_mmtm_ntu.py","file_url":"https://github.com/haamoon/mmtm/blob/HEAD/main_mmtm_ntu.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f5cbdca0600a0fd4"}},{"code_sha256_prefix":"880f16887cdc7f07","entry":"test_mmtm_track_acc","repo":"haamoon/mmtm","repo_kind":"listed","path":"main_mmtm_ntu.py","file_url":"https://github.com/haamoon/mmtm/blob/HEAD/main_mmtm_ntu.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"880f16887cdc7f07"}},{"code_sha256_prefix":"2d362aa3ef70c383","entry":"train_mmtm_track_acc","repo":"haamoon/mmtm","repo_kind":"listed","path":"main_mmtm_ntu.py","file_url":"https://github.com/haamoon/mmtm/blob/HEAD/main_mmtm_ntu.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2d362aa3ef70c383"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}