{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/multimodal-sentiment-analysis-using","title":"Multimodal Sentiment Analysis using Hierarchical Fusion with Context Modeling","arxiv_id":"1806.06228","date":"2018-06-16","proceeding":null,"authors":["N. Majumder","D. Hazarika","A. Gelbukh","E. Cambria","S. Poria"],"abstract":"Multimodal sentiment analysis is a very actively growing field of research. A\npromising area of opportunity in this field is to improve the multimodal fusion\nmechanism. We present a novel feature fusion strategy that proceeds in a\nhierarchical fashion, first fusing the modalities two in two and only then\nfusing all three modalities. On multimodal sentiment analysis of individual\nutterances, our strategy outperforms conventional concatenation of features by\n1%, which amounts to 5% reduction in error rate. On utterance-level multimodal\nsentiment analysis of multi-utterance video clips, for which current\nstate-of-the-art techniques incorporate contextual information from other\nutterances of the same clip, our hierarchical fusion gives up to 2.4% (almost\n10% error rate reduction) over currently used concatenation. The implementation\nof our method is publicly available in the form of open-source code.","url_abs":"http://arxiv.org/abs/1806.06228v1","url_pdf":"http://arxiv.org/pdf/1806.06228v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"multimodal-sentiment-analysis-using","repo_url":"https://github.com/SenticNet/hfusion","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":null}],"tasks":[{"task_slug":"multimodal-emotion-recognition","task_name":"Multimodal Emotion Recognition"},{"task_slug":"multimodal-sentiment-analysis","task_name":"Multimodal Sentiment Analysis"},{"task_slug":"sentiment-analysis","task_name":"Sentiment Analysis"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/multimodal-emotion-recognition-on-iemocap-4","task":"Multimodal Emotion Recognition","dataset":"IEMOCAP-4","model":"CHFusion","rank_in_archive_order":9,"of":11,"metrics":{"Accuracy":"76.5","F1":"76.8"},"uses_additional_data":false},{"leaderboard":"/sota/multimodal-sentiment-analysis-on-mosi","task":"Multimodal Sentiment Analysis","dataset":"MOSI","model":"CHFusion","rank_in_archive_order":11,"of":11,"metrics":{"Accuracy":"76.5%"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1806.06228","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}