{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/how-robust-r-u-evaluating-task-oriented","title":"\"How Robust r u?\": Evaluating Task-Oriented Dialogue Systems on Spoken Conversations","arxiv_id":"2109.13489","date":"2021-09-28","proceeding":null,"authors":["Seokhwan Kim","Yang Liu","Di Jin","Alexandros Papangelis","Karthik Gopalakrishnan","Behnam Hedayatnia","Dilek Hakkani-Tur"],"abstract":"Most prior work in dialogue modeling has been on written conversations mostly because of existing data sets. However, written dialogues are not sufficient to fully capture the nature of spoken conversations as well as the potential speech recognition errors in practical spoken dialogue systems. This work presents a new benchmark on spoken task-oriented conversations, which is intended to study multi-domain dialogue state tracking and knowledge-grounded dialogue modeling. We report that the existing state-of-the-art models trained on written conversations are not performing well on our spoken data, as expected. Furthermore, we observe improvements in task performances when leveraging n-best speech recognition hypotheses such as by combining predictions based on individual hypotheses. Our data set enables speech-based benchmarking of task-oriented dialogue systems.","url_abs":"https://arxiv.org/abs/2109.13489v1","url_pdf":"https://arxiv.org/pdf/2109.13489v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"how-robust-r-u-evaluating-task-oriented","repo_url":"https://github.com/alexa/alexa-with-dstc10-track2-dataset","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"dialogue-state-tracking","task_name":"Dialogue State Tracking"},{"task_slug":"multi-domain-dialogue-state-tracking","task_name":"Multi-domain Dialogue State Tracking"},{"task_slug":"speech-recognition","task_name":"Speech Recognition"},{"task_slug":"spoken-dialogue-systems","task_name":"Spoken Dialogue Systems"},{"task_slug":"task-oriented-dialogue-systems","task_name":"Task-Oriented Dialogue Systems"},{"task_slug":"speech-recognition-1","task_name":"speech-recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}