{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improving-robustness-of-neural-dialog-systems","title":"Improving Robustness of Neural Dialog Systems in a Data-Efficient Way with Turn Dropout","arxiv_id":"1811.12148","date":"2018-11-29","proceeding":null,"authors":["Igor Shalyminov","Sungjin Lee"],"abstract":"Neural network-based dialog models often lack robustness to anomalous,\nout-of-domain (OOD) user input which leads to unexpected dialog behavior and\nthus considerably limits such models' usage in mission-critical production\nenvironments. The problem is especially relevant in the setting of dialog\nsystem bootstrapping with limited training data and no access to OOD examples.\nIn this paper, we explore the problem of robustness of such systems to\nanomalous input and the associated to it trade-off in accuracies on seen and\nunseen data. We present a new dataset for studying the robustness of dialog\nsystems to OOD input, which is bAbI Dialog Task 6 augmented with OOD content in\na controlled way. We then present turn dropout, a simple yet efficient negative\nsampling-based technique for improving robustness of neural dialog models. We\ndemonstrate its effectiveness applied to Hybrid Code Network-family models\n(HCNs) which reach state-of-the-art results on our OOD-augmented dataset as\nwell as the original one. Specifically, an HCN trained with turn dropout\nachieves state-of-the-art performance of more than 75% per-utterance accuracy\non the augmented dataset's OOD turns and 74% F1-score as an OOD detector.\nFurthermore, we introduce a Variational HCN enhanced with turn dropout which\nachieves more than 56.5% accuracy on the original bAbI Task 6 dataset, thus\noutperforming the initially reported HCN's result.","url_abs":"http://arxiv.org/abs/1811.12148v1","url_pdf":"http://arxiv.org/pdf/1811.12148v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improving-robustness-of-neural-dialog-systems","repo_url":"https://github.com/ishalyminov/ood_robust_hcn","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[],"methods":[{"method_slug":"dropout","method_name":"Dropout"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1811.12148","atlas_url":"https://app.syntology.ai/?focus=1811.12148","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}