{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multimodal-sentiment-analysis-with-word-level","title":"Multimodal Sentiment Analysis with Word-Level Fusion and Reinforcement Learning","arxiv_id":"1802.00924","date":"2018-02-03","proceeding":null,"authors":["Minghai Chen","Sen Wang","Paul Pu Liang","Tadas Baltrušaitis","Amir Zadeh","Louis-Philippe Morency"],"abstract":"With the increasing popularity of video sharing websites such as YouTube and\nFacebook, multimodal sentiment analysis has received increasing attention from\nthe scientific community. Contrary to previous works in multimodal sentiment\nanalysis which focus on holistic information in speech segments such as bag of\nwords representations and average facial expression intensity, we develop a\nnovel deep architecture for multimodal sentiment analysis that performs\nmodality fusion at the word level. In this paper, we propose the Gated\nMultimodal Embedding LSTM with Temporal Attention (GME-LSTM(A)) model that is\ncomposed of 2 modules. The Gated Multimodal Embedding alleviates the\ndifficulties of fusion when there are noisy modalities. The LSTM with Temporal\nAttention performs word level fusion at a finer fusion resolution between input\nmodalities and attends to the most important time steps. As a result, the\nGME-LSTM(A) is able to better model the multimodal structure of speech through\ntime and perform better sentiment comprehension. We demonstrate the\neffectiveness of this approach on the publicly-available Multimodal Corpus of\nSentiment Intensity and Subjectivity Analysis (CMU-MOSI) dataset by achieving\nstate-of-the-art sentiment classification and regression results. Qualitative\nanalysis on our model emphasizes the importance of the Temporal Attention Layer\nin sentiment prediction because the additional acoustic and visual modalities\nare noisy. We also demonstrate the effectiveness of the Gated Multimodal\nEmbedding in selectively filtering these noisy modalities out. Our results and\nanalysis open new areas in the study of sentiment analysis in human\ncommunication and provide new models for multimodal fusion.","url_abs":"http://arxiv.org/abs/1802.00924v1","url_pdf":"http://arxiv.org/pdf/1802.00924v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multimodal-sentiment-analysis-with-word-level","repo_url":"https://github.com/clin366/MFN","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"multimodal-sentiment-analysis-with-word-level","repo_url":"https://github.com/pliang279/MFN","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"multimodal-sentiment-analysis","task_name":"Multimodal Sentiment Analysis"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"sentiment-analysis","task_name":"Sentiment Analysis"},{"task_slug":"sentiment-classification","task_name":"Sentiment Classification"},{"task_slug":"subjectivity-analysis","task_name":"Subjectivity Analysis"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1802.00924","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}