{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/exploiting-modality-invariant-feature-for","title":"Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalities","arxiv_id":"2210.15359","date":"2022-10-27","proceeding":null,"authors":["Haolin Zuo","Rui Liu","Jinming Zhao","Guanglai Gao","Haizhou Li"],"abstract":"Multimodal emotion recognition leverages complementary information across modalities to gain performance. However, we cannot guarantee that the data of all modalities are always present in practice. In the studies to predict the missing data across modalities, the inherent difference between heterogeneous modalities, namely the modality gap, presents a challenge. To address this, we propose to use invariant features for a missing modality imagination network (IF-MMIN) which includes two novel mechanisms: 1) an invariant feature learning strategy that is based on the central moment discrepancy (CMD) distance under the full-modality scenario; 2) an invariant feature based imagination module (IF-IM) to alleviate the modality gap during the missing modalities prediction, thus improving the robustness of multimodal joint representation. Comprehensive experiments on the benchmark dataset IEMOCAP demonstrate that the proposed model outperforms all baselines and invariantly improves the overall emotion recognition performance under uncertain missing-modality conditions. We release the code at: https://github.com/ZhuoYulang/IF-MMIN.","url_abs":"https://arxiv.org/abs/2210.15359v1","url_pdf":"https://arxiv.org/pdf/2210.15359v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"exploiting-modality-invariant-feature-for","repo_url":"https://github.com/zhuoyulang/if-mmin","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"emotion-recognition","task_name":"Emotion Recognition"},{"task_slug":"multimodal-emotion-recognition","task_name":"Multimodal Emotion Recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.15359","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.15359"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhuoyulang/if-mmin","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ee4151b9a0506771","entry":"build_array","repo":"zhuoyulang/if-mmin","repo_kind":"official","path":"draw.py","file_url":"https://github.com/zhuoyulang/if-mmin/blob/HEAD/draw.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ee4151b9a0506771"}},{"code_sha256_prefix":"0953c66f6d265ad1","entry":"masked_mean","repo":"zhuoyulang/if-mmin","repo_kind":"official","path":"models/MISA_model.py","file_url":"https://github.com/zhuoyulang/if-mmin/blob/HEAD/models/MISA_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0953c66f6d265ad1"}},{"code_sha256_prefix":"9da30be4666026a4","entry":"random_index","repo":"zhuoyulang/if-mmin","repo_kind":"official","path":"draw.py","file_url":"https://github.com/zhuoyulang/if-mmin/blob/HEAD/draw.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9da30be4666026a4"}},{"code_sha256_prefix":"e14ed990f333a627","entry":"read_pkl","repo":"zhuoyulang/if-mmin","repo_kind":"official","path":"draw.py","file_url":"https://github.com/zhuoyulang/if-mmin/blob/HEAD/draw.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e14ed990f333a627"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}