Multimodal Emotion Recognition

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

7 leaderboard tables shown for this task, 7 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

11 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

1 subtask in the archive's task tree.

Parent tasks archive 2025-07-28

Most implemented papers archive 2025-07-28

30 shown of 80 papers with code (180 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• Multimodal Speech Emotion Recognition and Ambiguity Resolution
  12 Apr 2019 5 repositories listed
  In this work, we adopt a feature-engineering based approach to tackle the task of speech emotion recognition.
• Multimodal Speech Emotion Recognition Using Audio and Text
  10 Oct 2018 4 repositories listed
  Speech emotion recognition is a challenging task, and extensive reliance has been placed on models that use audio features in building well-performing classifiers.
• MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learning
  18 Apr 2023 3 repositories listed
  The first Multimodal Emotion Recognition Challenge (MER 2023) was successfully held at ACM Multimedia.
• Complementary Fusion of Multi-Features and Multi-Modalities in Sentiment Analysis
  17 Apr 2019 3 repositories listed
  Therefore, in this paper, based on audio and text, we consider the task of multimodal sentiment analysis and propose a novel fusion strategy including both multi-feature fusion and multi-modality fusion to improve the…
• EmoVerse: Exploring Multimodal Large Language Models for Sentiment and Emotion Understanding
  11 Dec 2024 2 repositories listed
  While Multimodal Large Language Models (MLLMs) demonstrate robust general capabilities, they face considerable challenges in the field of affective computing, particularly in detecting subtle facial expressions and…
• Textualized and Feature-based Models for Compound Multimodal Emotion Recognition in the Wild
  17 Jul 2024 2 repositories listed Syntology ran 3 of 4 samples · 1 unverified · 4 pointer-only (licence)
  Our results indicate that multimodal textualization provides lower accuracy than feature-based models on C-EXPR-DB, where text transcripts are captured in the wild.
• Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning
  17 Jun 2024 2 repositories listed Syntology ran 1 of 2 samples · 1 unverified
  Accurate emotion perception is crucial for various applications, including human-computer interaction, education, and counseling.
• MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognition
  26 Apr 2024 2 repositories listed Syntology ran 5 of 9 samples · 4 unverified · 9 pointer-only (licence)
  However, this process may lead to inaccurate annotations, such as ignoring non-majority or non-candidate labels.
• COGMEN: COntextualized GNN based Multimodal Emotion recognitioN
  5 May 2022 2 repositories listed
  Emotions are an inherent part of human interactions, and consequently, it is imperative to develop AI systems that understand and recognize human emotions.
• Emotion Recognition in Audio and Video Using Deep Neural Networks
  15 Jun 2020 2 repositories listed
  Humans are able to comprehend information from multiple domains for e.
• DialogueRNN: An Attentive RNN for Emotion Detection in Conversations
  1 Nov 2018 2 repositories listed
  Emotion detection in conversations is a necessary step for a number of applications, including opinion mining over chat history, social media threads, debates, argumentation mining, understanding consumer feedback in…
• Multi-Modal Emotion recognition on IEMOCAP Dataset using Deep Learning
  16 Apr 2018 2 repositories listed
  Emotion recognition has become an important field of research in Human Computer Interactions as we improve upon the techniques for modelling the various aspects of behaviour.
• Context-Dependent Sentiment Analysis in User-Generated Videos
  1 Jul 2017 2 repositories listed
  Multimodal sentiment analysis is a developing area of research, which involves the identification of sentiments in videos.
• End-to-End Multimodal Emotion Recognition using Deep Neural Networks
  27 Apr 2017 2 repositories listed
  The system is then trained in an end-to-end fashion where - by also taking advantage of the correlations of the each of the streams - we manage to significantly outperform the traditional approaches based on auditory…
• Towards Robust Multimodal Emotion Recognition under Missing Modalities and Distribution Shifts
  12 Jun 2025 1 repository listed
  To address these issues, we propose a novel robust MER framework, Causal Inference Distiller (CIDer), and introduce a new task, Random Modality Feature Missing (RMFM), to generalize the definition of modality missing.
• TACFN: Transformer-based Adaptive Cross-modal Fusion Network for Multimodal Emotion Recognition
  10 May 2025 1 repository listed
  Specifically, for the redundant features, we make one modality perform intra-modal feature selection through a self-attention mechanism, so that the selected features can adaptively and efficiently interact with another…
• Feature-Based Dual Visual Feature Extraction Model for Compound Multimodal Emotion Recognition
  21 Mar 2025 1 repository listed
  This article presents our results for the eighth Affective Behavior Analysis in-the-wild (ABAW) competition.
• R1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcement Learning
  7 Mar 2025 1 repository listed Syntology ran 0 of 3 samples · 3 unverified · 3 pointer-only (licence)
  In this work, we present the first application of Reinforcement Learning with Verifiable Reward (RLVR) to an Omni-multimodal large language model in the context of emotion recognition, a task where both visual and audio…
• Latent Distribution Decoupling: A Probabilistic Framework for Uncertainty-Aware Multimodal Emotion Recognition
  19 Feb 2025 1 repository listed
  Specifically, we introduce a contrastive disentangled distribution mechanism within the emotion space to model the multimodal data, allowing for the extraction of semantic features and uncertainty.
• Milmer: a Framework for Multiple Instance Learning based Multimodal Emotion Recognition
  1 Feb 2025 1 repository listed
  Ablation studies further validate the contributions of each module, highlighting the significance of advanced feature extraction and fusion strategies in enhancing emotion recognition performance.
• SDR-GNN: Spectral Domain Reconstruction Graph Neural Network for Incomplete Multimodal Learning in Conversational Emotion Recognition
  29 Nov 2024 1 repository listed
  To address these issues, we propose a Spectral Domain Reconstruction Graph Neural Network (SDR-GNN) for incomplete multimodal learning in conversational emotion recognition.
• Hierarchical Hypercomplex Network for Multimodal Emotion Recognition
  13 Sep 2024 1 repository listed
  Then, a hypercomplex fusion module learns inter-modal relations among the embeddings of the different modalities.
• PHemoNet: A Multimodal Network for Physiological Signals
  13 Sep 2024 1 repository listed
  In this paper, we introduce PHemoNet, a fully hypercomplex network for multimodal emotion recognition from physiological signals.
• Recent Trends of Multimodal Affective Computing: A Survey from NLP Perspective
  11 Sep 2024 1 repository listed
  The goal of this survey is to explore the current landscape of multimodal affective research, identify development trends, and highlight the similarities and differences across various tasks, offering a comprehensive…
• Leveraging Contrastive Learning and Self-Training for Multimodal Emotion Recognition with Limited Labeled Samples
  23 Aug 2024 1 repository listed Syntology ran 1 of 1 samples · 0 unverified
  The Multimodal Emotion Recognition challenge MER2024 focuses on recognizing emotions using audio, language, and visual signals.
• SZTU-CMU at MER2024: Improving Emotion-LLaMA with Conv-Attention for Multimodal Emotion Recognition
  20 Aug 2024 1 repository listed Syntology ran 5 of 8 samples · 3 unverified
  This paper presents our winning approach for the MER-NOISE and MER-OV tracks of the MER2024 Challenge on multimodal emotion recognition.
• Enhancing Modal Fusion by Alignment and Label Matching for Multimodal Emotion Recognition
  18 Aug 2024 1 repository listed
  To address the limitation in multimodal emotion recognition (MER) performance arising from inter-modal information fusion, we propose a novel MER framework based on multitask learning where fusion occurs after…
• Multi Teacher Privileged Knowledge Distillation for Multimodal Expression Recognition
  16 Aug 2024 1 repository listed
  However, PKD methods based on structural similarity are primarily confined to learning from a single joint teacher representation, which limits their robustness, accuracy, and ability to learn from diverse multimodal…
• Tracing Intricate Cues in Dialogue: Joint Graph Structure and Sentiment Dynamics for Multimodal Emotion Recognition
  31 Jul 2024 1 repository listed
  Furthermore, GraphSmile is effortlessly applied to multimodal sentiment analysis in conversation (MSAC), forging a unified multimodal affective model capable of executing MERC and MSAC tasks.
• Multimodal Emotion Recognition using Audio-Video Transformer Fusion with Cross Attention
  26 Jul 2024 1 repository listed
  Despite its potential, multimodal emotion recognition faces significant challenges, particularly in synchronization, feature extraction, and fusion of diverse data sources.

Syntology lines on 6 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.