{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/emowoz-a-large-scale-corpus-and-labelling","title":"EmoWOZ: A Large-Scale Corpus and Labelling Scheme for Emotion Recognition in Task-Oriented Dialogue Systems","arxiv_id":"2109.04919","date":"2021-09-10","proceeding":"LREC 2022 6","authors":["Shutong Feng","Nurul Lubis","Christian Geishauser","Hsien-Chin Lin","Michael Heck","Carel van Niekerk","Milica Gašić"],"abstract":"The ability to recognise emotions lends a conversational artificial intelligence a human touch. While emotions in chit-chat dialogues have received substantial attention, emotions in task-oriented dialogues remain largely unaddressed. This is despite emotions and dialogue success having equally important roles in a natural system. Existing emotion-annotated task-oriented corpora are limited in size, label richness, and public availability, creating a bottleneck for downstream tasks. To lay a foundation for studies on emotions in task-oriented dialogues, we introduce EmoWOZ, a large-scale manually emotion-annotated corpus of task-oriented dialogues. EmoWOZ is based on MultiWOZ, a multi-domain task-oriented dialogue dataset. It contains more than 11K dialogues with more than 83K emotion annotations of user utterances. In addition to Wizard-of-Oz dialogues from MultiWOZ, we collect human-machine dialogues within the same set of domains to sufficiently cover the space of various emotions that can happen during the lifetime of a data-driven dialogue system. To the best of our knowledge, this is the first large-scale open-source corpus of its kind. We propose a novel emotion labelling scheme, which is tailored to task-oriented dialogues. We report a set of experimental results to show the usability of this corpus for emotion recognition and state tracking in task-oriented dialogues.","url_abs":"https://arxiv.org/abs/2109.04919v2","url_pdf":"https://arxiv.org/pdf/2109.04919v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"emowoz-a-large-scale-corpus-and-labelling","repo_url":"https://gitlab.cs.uni-duesseldorf.de/general/dsml/emowoz-public","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"emotion-recognition","task_name":"Emotion Recognition"},{"task_slug":"emotion-recognition-in-conversation","task_name":"Emotion Recognition in Conversation"},{"task_slug":"task-oriented-dialogue-systems","task_name":"Task-Oriented Dialogue Systems"}],"methods":[],"datasets_introduced":[{"slug":"emowoz-1","name":"EmoWOZ","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/emotion-recognition-in-conversation-on-emowoz","task":"Emotion Recognition in Conversation","dataset":"EmoWoz","model":"COSMIC","rank_in_archive_order":1,"of":6,"metrics":{"Macro F1":"61.12","Macro F1 (w/o Neutral)":"56.34","Weighted F1":"85.94","Weighted F1 (w/o Neutral)":"77.09"},"uses_additional_data":false},{"leaderboard":"/sota/emotion-recognition-in-conversation-on-emowoz","task":"Emotion Recognition in Conversation","dataset":"EmoWoz","model":"ContextBERT","rank_in_archive_order":2,"of":6,"metrics":{"Macro F1":"59.79","Macro F1 (w/o Neutral)":"54.30","Weighted F1":"88.33","Weighted F1 (w/o Neutral)":"79.67"},"uses_additional_data":false},{"leaderboard":"/sota/emotion-recognition-in-conversation-on-emowoz","task":"Emotion Recognition in Conversation","dataset":"EmoWoz","model":"DialogueRNN-BERT","rank_in_archive_order":3,"of":6,"metrics":{"Macro F1":"57.10","Macro F1 (w/o Neutral)":"52.15","Weighted F1":"83.41","Weighted F1 (w/o Neutral)":"75.50"},"uses_additional_data":false},{"leaderboard":"/sota/emotion-recognition-in-conversation-on-emowoz","task":"Emotion Recognition in Conversation","dataset":"EmoWoz","model":"BERT","rank_in_archive_order":4,"of":6,"metrics":{"Macro F1":"55.80","Macro F1 (w/o Neutral)":"50.14","Weighted F1":"84.83","Weighted F1 (w/o Neutral)":"73.55"},"uses_additional_data":false},{"leaderboard":"/sota/emotion-recognition-in-conversation-on-emowoz","task":"Emotion Recognition in Conversation","dataset":"EmoWoz","model":"DialogueRNN-GloVe","rank_in_archive_order":5,"of":6,"metrics":{"Macro F1":"46.33","Macro F1 (w/o Neutral)":"40.14","Weighted F1":"80.76","Weighted F1 (w/o Neutral)":"74.56"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2109.04919","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}