{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/what-is-wrong-with-you-leveraging-user","title":"What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation","arxiv_id":"2203.13927","date":"2022-03-25","proceeding":"Findings (ACL) 2022 5","authors":["Sarik Ghazarian","Behnam Hedayatnia","Alexandros Papangelis","Yang Liu","Dilek Hakkani-Tur"],"abstract":"Accurate automatic evaluation metrics for open-domain dialogs are in high demand. Existing model-based metrics for system response evaluation are trained on human annotated data, which is cumbersome to collect. In this work, we propose to use information that can be automatically extracted from the next user utterance, such as its sentiment or whether the user explicitly ends the conversation, as a proxy to measure the quality of the previous system response. This allows us to train on a massive set of dialogs with weak supervision, without requiring manual system turn quality annotations. Experiments show that our model is comparable to models trained on human annotated data. Furthermore, our model generalizes across both spoken and written open-domain dialog corpora collected from real and paid users.","url_abs":"https://arxiv.org/abs/2203.13927v1","url_pdf":"https://arxiv.org/pdf/2203.13927v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"what-is-wrong-with-you-leveraging-user","repo_url":"https://github.com/alexa/conture","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"dialogue-evaluation","task_name":"Dialogue Evaluation"},{"task_slug":"open-domain-dialog","task_name":"Open-Domain Dialog"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2203.13927","atlas_url":"https://app.syntology.ai/?focus=2203.13927","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}