Papers › Multimodal Emotion Recognition on RAVDESS Dataset Using Transfer Learning
Multimodal Emotion Recognition on RAVDESS Dataset Using Transfer Learning
Cristina Luna-Jiménez, David Griol, Zoraida Callejas, Ricardo Kleinlein, Juan M. Montero, Fernando Fernández-Martínez
Emotion Recognition is attracting the attention of the research community due to the multiple areas where it can be applied, such as in healthcare or in road safety systems. In this paper, we propose a multimodal emotion recognition system that relies on speech and facial information. For the speech-based modality, we evaluated several transfer-learning techniques, more specifically, embedding extraction and Fine-Tuning. The best accuracy results were achieved when we fine-tuned the CNN-14 of the PANNs framework, confirming that the training was more robust when it did not start from scratch and the tasks were similar. Regarding the facial emotion recognizers, we propose a framework that consists of a pre-trained Spatial Transformer Network on saliency maps and facial images followed by a bi-LSTM with an attention mechanism. The error analysis reported that the frame-based systems could present some problems when they were used directly to solve a video-based task despite the domain adaptation, which opens a new line of research to discover new ways to correct this mismatch and take advantage of the embedded knowledge of these pre-trained models. Finally, from the combination of these two modalities with a late fusion strategy, we achieved 80.08% accuracy on the RAVDESS dataset on a subject-wise 5-CV evaluation, classifying eight emotions. The results revealed that these modalities carry relevant information to detect users’ emotional state and their combination enables improvement of system performance.
Code
No code repository is listed for this paper in the archive or in Syntology's graph.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Emotion Recognition | RAVDESS | Logistic Regression on posteriors of the CNN-14&biLSTM-GuidedST | Accuracy | 80.08% | #3 of 5 | Archive leaderboard | report |
| Facial Emotion Recognition | RAVDESS | Guided-ST and bi-LSTM with attention | Accuracy | 57.08% | #4 of 4 | Archive leaderboard | report |
| Speech Emotion Recognition | RAVDESS | CNN-14 (Fine-Tuning) | Accuracy | 76.58% | #4 of 5 | Archive leaderboard | report |
| Speech Emotion Recognition | RAVDESS | AlexNet (FineTuning) | Accuracy | 61.67% | #5 of 5 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections