{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improving-multimodal-fusion-with-hierarchical","title":"Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis","arxiv_id":"2109.00412","date":"2021-09-01","proceeding":"EMNLP 2021 11","authors":["Wei Han","Hui Chen","Soujanya Poria"],"abstract":"In multimodal sentiment analysis (MSA), the performance of a model highly depends on the quality of synthesized embeddings. These embeddings are generated from the upstream process called multimodal fusion, which aims to extract and combine the input unimodal raw data to produce a richer multimodal representation. Previous work either back-propagates the task loss or manipulates the geometric property of feature spaces to produce favorable fusion results, which neglects the preservation of critical task-related information that flows from input to the fusion results. In this work, we propose a framework named MultiModal InfoMax (MMIM), which hierarchically maximizes the Mutual Information (MI) in unimodal input pairs (inter-modality) and between multimodal fusion result and unimodal input in order to maintain task-related information through multimodal fusion. The framework is jointly trained with the main task (MSA) to improve the performance of the downstream MSA task. To address the intractable issue of MI bounds, we further formulate a set of computationally simple parametric and non-parametric methods to approximate their truth value. Experimental results on the two widely used datasets demonstrate the efficacy of our approach. The implementation of this work is publicly available at https://github.com/declare-lab/Multimodal-Infomax.","url_abs":"https://arxiv.org/abs/2109.00412v2","url_pdf":"https://arxiv.org/pdf/2109.00412v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improving-multimodal-fusion-with-hierarchical","repo_url":"https://github.com/declare-lab/multimodal-deep-learning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"improving-multimodal-fusion-with-hierarchical","repo_url":"https://github.com/declare-lab/multimodal-infomax","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"multimodal-sentiment-analysis","task_name":"Multimodal Sentiment Analysis"},{"task_slug":"sentiment-analysis","task_name":"Sentiment Analysis"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/multimodal-sentiment-analysis-on-cmu-mosi","task":"Multimodal Sentiment Analysis","dataset":"CMU-MOSI","model":"MMIM","rank_in_archive_order":7,"of":12,"metrics":{"Acc-2":"84.14","Acc-7":"46.65","Corr":"0.8","F1":"84","MAE":"0.7"},"uses_additional_data":false},{"leaderboard":"/sota/multimodal-sentiment-analysis-on-cmu-mosi","task":"Multimodal Sentiment Analysis","dataset":"CMU-MOSI","model":"self-M","rank_in_archive_order":8,"of":12,"metrics":{"Acc-2":"82.54","Acc-7":"45.79","Corr":"0.795","F1":"82.68","MAE":"0.712"},"uses_additional_data":false},{"leaderboard":"/sota/multimodal-sentiment-analysis-on-cmu-mosi","task":"Multimodal Sentiment Analysis","dataset":"CMU-MOSI","model":"MAG-BERT*","rank_in_archive_order":9,"of":12,"metrics":{"Acc-2":"82.37","Acc-7":"43.62","Corr":"0.781","F1":"82.5","MAE":"0.727"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2109.00412","atlas_url":"https://app.syntology.ai/?focus=2109.00412","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2109.00412"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/declare-lab/multimodal-infomax","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/declare-lab/multimodal-deep-learning","reach":null}],"summary":{"ran":4,"unverified":3},"by_repo_kind":{"official":{"samples":7,"ran":4,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5ca7b9a6d3b4c3cc","entry":"CPC","repo":"declare-lab/multimodal-infomax","repo_kind":"official","path":"src/model.py","file_url":"https://github.com/declare-lab/multimodal-infomax/blob/HEAD/src/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5ca7b9a6d3b4c3cc"}},{"code_sha256_prefix":"ada546af609682c0","entry":"MMILB","repo":"declare-lab/multimodal-infomax","repo_kind":"official","path":"src/model.py","file_url":"https://github.com/declare-lab/multimodal-infomax/blob/HEAD/src/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ada546af609682c0"}},{"code_sha256_prefix":"cc8afd7ca4697719","entry":"RNNEncoder","repo":"declare-lab/multimodal-infomax","repo_kind":"official","path":"src/model.py","file_url":"https://github.com/declare-lab/multimodal-infomax/blob/HEAD/src/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cc8afd7ca4697719"}},{"code_sha256_prefix":"5b3fe6fc63268031","entry":"SubNet","repo":"declare-lab/multimodal-infomax","repo_kind":"official","path":"src/model.py","file_url":"https://github.com/declare-lab/multimodal-infomax/blob/HEAD/src/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5b3fe6fc63268031"}},{"code_sha256_prefix":"c96cd058607bb4ec","entry":"LanguageEmbeddingLayer","repo":"declare-lab/multimodal-infomax","repo_kind":"official","path":"src/model.py","file_url":"https://github.com/declare-lab/multimodal-infomax/blob/HEAD/src/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c96cd058607bb4ec"}},{"code_sha256_prefix":"660f8740ef5c0c4d","entry":"MISA","repo":"declare-lab/multimodal-deep-learning","repo_kind":"official","path":"MISA/src/models.py","file_url":"https://github.com/declare-lab/multimodal-deep-learning/blob/HEAD/MISA/src/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"660f8740ef5c0c4d"}},{"code_sha256_prefix":"d665c7a7fd583f1c","entry":"MMIM","repo":"declare-lab/multimodal-infomax","repo_kind":"official","path":"src/model.py","file_url":"https://github.com/declare-lab/multimodal-infomax/blob/HEAD/src/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d665c7a7fd583f1c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}