{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multimodal-emotion-recognition-with-high","title":"Multimodal Emotion Recognition with High-level Speech and Text Features","arxiv_id":"2111.10202","date":"2021-09-29","proceeding":null,"authors":["Mariana Rodrigues Makiuchi","Kuniaki Uto","Koichi Shinoda"],"abstract":"Automatic emotion recognition is one of the central concerns of the Human-Computer Interaction field as it can bridge the gap between humans and machines. Current works train deep learning models on low-level data representations to solve the emotion recognition task. Since emotion datasets often have a limited amount of data, these approaches may suffer from overfitting, and they may learn based on superficial cues. To address these issues, we propose a novel cross-representation speech model, inspired by disentanglement representation learning, to perform emotion recognition on wav2vec 2.0 speech features. We also train a CNN-based model to recognize emotions from text features extracted with Transformer-based models. We further combine the speech-based and text-based results with a score fusion approach. Our method is evaluated on the IEMOCAP dataset in a 4-class classification problem, and it surpasses current works on speech-only, text-only, and multimodal emotion recognition.","url_abs":"https://arxiv.org/abs/2111.10202v1","url_pdf":"https://arxiv.org/pdf/2111.10202v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multimodal-emotion-recognition-with-high","repo_url":"https://github.com/mmakiuchi/multimodal_emotion_recognition","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"disentanglement","task_name":"Disentanglement"},{"task_slug":"emotion-recognition","task_name":"Emotion Recognition"},{"task_slug":"multimodal-emotion-recognition","task_name":"Multimodal Emotion Recognition"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"high","task_name":"Vocal Bursts Intensity Prediction"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2111.10202","atlas_url":"https://app.syntology.ai/?focus=2111.10202","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}