Papers › Multimodal Prompt Transformer with Hybrid Contrastive Learning for Emotion Recognition...
Multimodal Prompt Transformer with Hybrid Contrastive Learning for Emotion Recognition in Conversation
Shihao Zou, Xianying Huang, Xudong Shen
Emotion Recognition in Conversation (ERC) plays an important role in driving the development of human-machine interaction. Emotions can exist in multiple modalities, and multimodal ERC mainly faces two problems: (1) the noise problem in the cross-modal information fusion process, and (2) the prediction problem of less sample emotion labels that are semantically similar but different categories. To address these issues and fully utilize the features of each modality, we adopted the following strategies: first, deep emotion cues extraction was performed on modalities with strong representation ability, and feature filters were designed as multimodal prompt information for modalities with weak representation ability. Then, we designed a Multimodal Prompt Transformer (MPT) to perform cross-modal information fusion. MPT embeds multimodal fusion information into each attention layer of the Transformer, allowing prompt information to participate in encoding textual features and being fused with multi-level textual information to obtain better multimodal fusion features. Finally, we used the Hybrid Contrastive Learning (HCL) strategy to optimize the model's ability to handle labels with few samples. This strategy uses unsupervised contrastive learning to improve the representation ability of multimodal fusion and supervised contrastive learning to mine the information of labels with few samples. Experimental results show that our proposed model outperforms state-of-the-art models in ERC on two benchmark datasets.
Code
No code repository is listed for this paper in the archive or in Syntology's graph.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Emotion Recognition in Conversation | IEMOCAP | MPT-HCL | Accuracy | 72.83 | #4 of 59 | Archive leaderboard | report |
| Emotion Recognition in Conversation | IEMOCAP | MPT-HCL | Weighted-F1 | 72.51 | #4 of 59 | Archive leaderboard | report |
| Emotion Recognition in Conversation | MELD | MPT-HCL | Accuracy | 65.86 | #36 of 68 | Archive leaderboard | report |
| Emotion Recognition in Conversation | MELD | MPT-HCL | Weighted-F1 | 65.02 | #36 of 68 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections