{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/joyful-joint-modality-fusion-and-graph","title":"Joyful: Joint Modality Fusion and Graph Contrastive Learning for Multimodal Emotion Recognition","arxiv_id":"2311.11009","date":"2023-11-18","proceeding":null,"authors":["Dongyuan Li","Yusong Wang","Kotaro Funakoshi","Manabu Okumura"],"abstract":"Multimodal emotion recognition aims to recognize emotions for each utterance of multiple modalities, which has received increasing attention for its application in human-machine interaction. Current graph-based methods fail to simultaneously depict global contextual features and local diverse uni-modal features in a dialogue. Furthermore, with the number of graph layers increasing, they easily fall into over-smoothing. In this paper, we propose a method for joint modality fusion and graph contrastive learning for multimodal emotion recognition (Joyful), where multimodality fusion, contrastive learning, and emotion recognition are jointly optimized. Specifically, we first design a new multimodal fusion mechanism that can provide deep interaction and fusion between the global contextual and uni-modal specific features. Then, we introduce a graph contrastive learning framework with inter-view and intra-view contrastive losses to learn more distinguishable representations for samples with different sentiments. Extensive experiments on three benchmark datasets indicate that Joyful achieved state-of-the-art (SOTA) performance compared to all baselines.","url_abs":"https://arxiv.org/abs/2311.11009v1","url_pdf":"https://arxiv.org/pdf/2311.11009v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"joyful-joint-modality-fusion-and-graph","repo_url":"https://github.com/wykstc/MERC-main","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"emotion-recognition","task_name":"Emotion Recognition"},{"task_slug":"emotion-recognition-in-conversation","task_name":"Emotion Recognition in Conversation"},{"task_slug":"face-swapping","task_name":"Face Swapping"},{"task_slug":"multimodal-emotion-recognition","task_name":"Multimodal Emotion Recognition"}],"methods":[{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"},{"method_slug":"graph-contrastive-coding","method_name":"Graph Contrastive Coding"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/emotion-recognition-in-conversation-on-7","task":"Emotion Recognition in Conversation","dataset":"IEMOCAP-4","model":"Joyful","rank_in_archive_order":2,"of":8,"metrics":{"Weighted F1":"85.70"},"uses_additional_data":false},{"leaderboard":"/sota/face-swapping-on-hod","task":"Face Swapping","dataset":"HOD","model":"Work","rank_in_archive_order":1,"of":1,"metrics":{"0-shot MRR":"Good"},"uses_additional_data":true},{"leaderboard":"/sota/multimodal-emotion-recognition-on-iemocap","task":"Multimodal Emotion Recognition","dataset":"IEMOCAP","model":"Joyful","rank_in_archive_order":2,"of":2,"metrics":{"Accuracy":"71.0","Weighted F1":"70.50"},"uses_additional_data":false},{"leaderboard":"/sota/multimodal-emotion-recognition-on-iemocap-4","task":"Multimodal Emotion Recognition","dataset":"IEMOCAP-4","model":"Joyful","rank_in_archive_order":2,"of":11,"metrics":{"Accuracy":"85.60","Weighted F1":"85.70"},"uses_additional_data":false},{"leaderboard":"/sota/multimodal-emotion-recognition-on-meld","task":"Multimodal Emotion Recognition","dataset":"MELD","model":"Joyful","rank_in_archive_order":3,"of":3,"metrics":{"Accuracy":"62.53","Weighted F1":"61.77"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}