{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mmoe-mixture-of-multimodal-interaction","title":"MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts","arxiv_id":"2311.09580","date":"2023-11-16","proceeding":null,"authors":["Haofei Yu","Zhengyang Qi","Lawrence Jang","Ruslan Salakhutdinov","Louis-Philippe Morency","Paul Pu Liang"],"abstract":"Advances in multimodal models have greatly improved how interactions relevant to various tasks are modeled. Today's multimodal models mainly focus on the correspondence between images and text, using this for tasks like image-text matching. However, this covers only a subset of real-world interactions. Novel interactions, such as sarcasm expressed through opposing spoken words and gestures or humor expressed through utterances and tone of voice, remain challenging. In this paper, we introduce an approach to enhance multimodal models, which we call Multimodal Mixtures of Experts (MMoE). The key idea in MMoE is to train separate expert models for each type of multimodal interaction, such as redundancy present in both modalities, uniqueness in one modality, or synergy that emerges when both modalities are fused. On a sarcasm detection task (MUStARD) and a humor detection task (URFUNNY), we obtain new state-of-the-art results. MMoE is also able to be applied to various types of models to gain improvement.","url_abs":"https://arxiv.org/abs/2311.09580v3","url_pdf":"https://arxiv.org/pdf/2311.09580v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mmoe-mixture-of-multimodal-interaction","repo_url":"https://github.com/lwaekfjlk/mmoe","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"binary-classification","task_name":"Binary Classification"},{"task_slug":"descriptive","task_name":"Descriptive"},{"task_slug":"humor-detection","task_name":"Humor Detection"},{"task_slug":"image-text-matching","task_name":"Image-text matching"},{"task_slug":"sarcasm-detection","task_name":"Sarcasm Detection"},{"task_slug":"text-matching","task_name":"Text Matching"},{"task_slug":"multimodal-interaction","task_name":"multimodal interaction"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2311.09580","atlas_url":"https://app.syntology.ai/?focus=2311.09580","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.09580"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lwaekfjlk/mmoe","reach":{"status":"ok"}}],"summary":{"ran":4,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"5a68505c76d0d6ad","entry":"collate_fn","repo":"lwaekfjlk/mmoe","repo_kind":"official","path":"data_gen_image_description_CogVLM2/utils.py","file_url":"https://github.com/lwaekfjlk/mmoe/blob/HEAD/data_gen_image_description_CogVLM2/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5a68505c76d0d6ad"}},{"code_sha256_prefix":"fe17eaee8c324b71","entry":"get_prediction","repo":"lwaekfjlk/mmoe","repo_kind":"official","path":"data_gen_text_label_Qwen2/utils.py","file_url":"https://github.com/lwaekfjlk/mmoe/blob/HEAD/data_gen_text_label_Qwen2/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe17eaee8c324b71"}},{"code_sha256_prefix":"80585f881584d654","entry":"recur_move_to","repo":"lwaekfjlk/mmoe","repo_kind":"official","path":"data_gen_image_description_CogVLM2/utils.py","file_url":"https://github.com/lwaekfjlk/mmoe/blob/HEAD/data_gen_image_description_CogVLM2/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"80585f881584d654"}},{"code_sha256_prefix":"654781a37d6e0443","entry":"select_top_percent_as_one","repo":"lwaekfjlk/mmoe","repo_kind":"official","path":"data_gen_text_label_Qwen2/utils.py","file_url":"https://github.com/lwaekfjlk/mmoe/blob/HEAD/data_gen_text_label_Qwen2/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"654781a37d6e0443"}},{"code_sha256_prefix":"39a0806f57f48a3c","entry":"prompt_llm","repo":"lwaekfjlk/mmoe","repo_kind":"official","path":"data_gen_text_label_Qwen2/utils.py","file_url":"https://github.com/lwaekfjlk/mmoe/blob/HEAD/data_gen_text_label_Qwen2/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39a0806f57f48a3c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}