{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hierarchical-fusion-for-online-multimodal","title":"Hierarchical Fusion for Online Multimodal Dialog Act Classification","arxiv_id":null,"date":"2023-12-08","proceeding":"EMNLP 2023 12","authors":["Md Messal Monem Miah","Adarsh Pyarelal","Ruihong Huang"],"abstract":"We propose a framework for online multimodal dialog act (DA) classification based on raw audio and ASR-generated transcriptions of current and past utterances. Existing multimodal DA classification approaches are limited by ineffective audio modeling and late-stage fusion. We showcase significant improvements in multimodal DA classification by integrating modalities at a more granular level and incorporating recent advancements in large language and audio models for audio feature extraction. We further investigate the effectiveness of self-attention and cross-attention mechanisms in modeling utterances and dialogs for DA classification. We achieve a substantial increase of 3 percentage points in the F1 score relative to current state-of-the-art models on two prominent DA classification datasets, MRDA and EMOTyDA.","url_abs":"https://aclanthology.org/2023.findings-emnlp.505.pdf","url_pdf":"https://aclanthology.org/2023.findings-emnlp.505.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hierarchical-fusion-for-online-multimodal","repo_url":"https://github.com/Dipto084/Hierarchical-Fusion-for-Online-Multimodal-Dialog-Act-Classification","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"dialog-act-classification","task_name":"Dialog Act Classification"},{"task_slug":"dialogue-act-classification","task_name":"Dialogue Act Classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/dialogue-act-classification-on-emotyda","task":"Dialogue Act Classification","dataset":"EMOTyDA","model":"Hierarchical Fusion","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy":"63.42"},"uses_additional_data":false},{"leaderboard":"/sota/dialogue-act-classification-on-icsi-meeting","task":"Dialogue Act Classification","dataset":"ICSI Meeting Recorder Dialog Act (MRDA) corpus","model":"Hierarchical Fusion","rank_in_archive_order":2,"of":8,"metrics":{"Accuracy":"91.8"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}