{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-fine-line-between-linguistic","title":"The Fine Line between Linguistic Generalization and Failure in Seq2Seq-Attention Models","arxiv_id":"1805.01445","date":"2018-05-03","proceeding":"WS 2018 6","authors":["Noah Weber","Leena Shekhar","Niranjan Balasubramanian"],"abstract":"Seq2Seq based neural architectures have become the go-to architecture to\napply to sequence to sequence language tasks. Despite their excellent\nperformance on these tasks, recent work has noted that these models usually do\nnot fully capture the linguistic structure required to generalize beyond the\ndense sections of the data distribution \\cite{ettinger2017towards}, and as\nsuch, are likely to fail on samples from the tail end of the distribution (such\nas inputs that are noisy \\citep{belkinovnmtbreak} or of different lengths\n\\citep{bentivoglinmtlength}). In this paper, we look at a model's ability to\ngeneralize on a simple symbol rewriting task with a clearly defined structure.\nWe find that the model's ability to generalize this structure beyond the\ntraining distribution depends greatly on the chosen random seed, even when\nperformance on the standard test set remains the same. This suggests that a\nmodel's ability to capture generalizable structure is highly sensitive.\nMoreover, this sensitivity may not be apparent when evaluating it on standard\ntest sets.","url_abs":"http://arxiv.org/abs/1805.01445v2","url_pdf":"http://arxiv.org/pdf/1805.01445v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-fine-line-between-linguistic","repo_url":"https://github.com/LeenaShekhar/FailureAndGeneralizationDataset","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"the-fine-line-between-linguistic","repo_url":"https://github.com/i-machine-think/machine-tasks","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1805.01445","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}