{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/misa-modality-invariant-and-specific","title":"MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis","arxiv_id":"2005.03545","date":"2020-05-07","proceeding":null,"authors":["Devamanyu Hazarika","Roger Zimmermann","Soujanya Poria"],"abstract":"Multimodal Sentiment Analysis is an active area of research that leverages multimodal signals for affective understanding of user-generated videos. The predominant approach, addressing this task, has been to develop sophisticated fusion techniques. However, the heterogeneous nature of the signals creates distributional modality gaps that pose significant challenges. In this paper, we aim to learn effective modality representations to aid the process of fusion. We propose a novel framework, MISA, which projects each modality to two distinct subspaces. The first subspace is modality-invariant, where the representations across modalities learn their commonalities and reduce the modality gap. The second subspace is modality-specific, which is private to each modality and captures their characteristic features. These representations provide a holistic view of the multimodal data, which is used for fusion that leads to task predictions. Our experiments on popular sentiment analysis benchmarks, MOSI and MOSEI, demonstrate significant gains over state-of-the-art models. We also consider the task of Multimodal Humor Detection and experiment on the recently proposed UR_FUNNY dataset. Here too, our model fares better than strong baselines, establishing MISA as a useful multimodal framework.","url_abs":"https://arxiv.org/abs/2005.03545v3","url_pdf":"https://arxiv.org/pdf/2005.03545v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"misa-modality-invariant-and-specific","repo_url":"https://github.com/declare-lab/MISA","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"misa-modality-invariant-and-specific","repo_url":"https://github.com/declare-lab/multimodal-deep-learning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"humor-detection","task_name":"Humor Detection"},{"task_slug":"multimodal-sentiment-analysis","task_name":"Multimodal Sentiment Analysis"},{"task_slug":"sentiment-analysis","task_name":"Sentiment Analysis"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2005.03545","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2005.03545"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/declare-lab/MISA","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/declare-lab/multimodal-deep-learning","reach":null}],"summary":{"ran_draft_wrong":1,"ran_violates":1,"unverified":8},"by_repo_kind":{"official":{"samples":10,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"70db7c9cbe9e680d","entry":"load_pickle","repo":"declare-lab/MISA","repo_kind":"official","path":"src/create_dataset.py","file_url":"https://github.com/declare-lab/MISA/blob/HEAD/src/create_dataset.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"70db7c9cbe9e680d"}},{"code_sha256_prefix":"38096a698c1dafa9","entry":"str2bool","repo":"declare-lab/MISA","repo_kind":"official","path":"src/config.py","file_url":"https://github.com/declare-lab/MISA/blob/HEAD/src/config.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"38096a698c1dafa9"}},{"code_sha256_prefix":"e5bfca4621471280","entry":"base_time_desc_decorator","repo":"declare-lab/MISA","repo_kind":"official","path":"src/utils/time_track.py","file_url":"https://github.com/declare-lab/MISA/blob/HEAD/src/utils/time_track.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e5bfca4621471280"}},{"code_sha256_prefix":"c534599cc75f0b11","entry":"get_config","repo":"declare-lab/MISA","repo_kind":"official","path":"src/config.py","file_url":"https://github.com/declare-lab/MISA/blob/HEAD/src/config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c534599cc75f0b11"}},{"code_sha256_prefix":"c4b17078505c0346","entry":"load_emb","repo":"declare-lab/MISA","repo_kind":"official","path":"src/create_dataset.py","file_url":"https://github.com/declare-lab/MISA/blob/HEAD/src/create_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c4b17078505c0346"}},{"code_sha256_prefix":"83f6167b0c8caf55","entry":"masked_max","repo":"declare-lab/MISA","repo_kind":"official","path":"src/models.py","file_url":"https://github.com/declare-lab/MISA/blob/HEAD/src/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"83f6167b0c8caf55"}},{"code_sha256_prefix":"0953c66f6d265ad1","entry":"masked_mean","repo":"declare-lab/MISA","repo_kind":"official","path":"src/models.py","file_url":"https://github.com/declare-lab/MISA/blob/HEAD/src/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0953c66f6d265ad1"}},{"code_sha256_prefix":"19017e8c3e1ed55b","entry":"time_desc_decorator","repo":"declare-lab/MISA","repo_kind":"official","path":"src/utils/time_track.py","file_url":"https://github.com/declare-lab/MISA/blob/HEAD/src/utils/time_track.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"19017e8c3e1ed55b"}},{"code_sha256_prefix":"1daaefb0404fa995","entry":"to_cpu","repo":"declare-lab/MISA","repo_kind":"official","path":"src/utils/convert.py","file_url":"https://github.com/declare-lab/MISA/blob/HEAD/src/utils/convert.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1daaefb0404fa995"}},{"code_sha256_prefix":"abad03534eaa9fa7","entry":"to_gpu","repo":"declare-lab/MISA","repo_kind":"official","path":"src/utils/convert.py","file_url":"https://github.com/declare-lab/MISA/blob/HEAD/src/utils/convert.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"abad03534eaa9fa7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}