{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/assessing-the-ability-of-lstms-to-learn","title":"Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies","arxiv_id":"1611.01368","date":"2016-11-04","proceeding":"TACL 2016 1","authors":["Tal Linzen","Emmanuel Dupoux","Yoav Goldberg"],"abstract":"The success of long short-term memory (LSTM) neural networks in language\nprocessing is typically attributed to their ability to capture long-distance\nstatistical regularities. Linguistic regularities are often sensitive to\nsyntactic structure; can such dependencies be captured by LSTMs, which do not\nhave explicit structural representations? We begin addressing this question\nusing number agreement in English subject-verb dependencies. We probe the\narchitecture's grammatical competence both using training objectives with an\nexplicit grammatical target (number prediction, grammaticality judgments) and\nusing language models. In the strongly supervised settings, the LSTM achieved\nvery high overall accuracy (less than 1% errors), but errors increased when\nsequential and structural information conflicted. The frequency of such errors\nrose sharply in the language-modeling setting. We conclude that LSTMs can\ncapture a non-trivial amount of grammatical structure given targeted\nsupervision, but stronger architectures may be required to further reduce\nerrors; furthermore, the language modeling signal is insufficient for capturing\nsyntax-sensitive dependencies, and should be supplemented with more direct\nsupervision if such dependencies need to be captured.","url_abs":"http://arxiv.org/abs/1611.01368v1","url_pdf":"http://arxiv.org/pdf/1611.01368v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"assessing-the-ability-of-lstms-to-learn","repo_url":"https://github.com/Hritikbansal/RNNs_SVA_OOD","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"assessing-the-ability-of-lstms-to-learn","repo_url":"https://github.com/huggingface/bert-syntax","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"assessing-the-ability-of-lstms-to-learn","repo_url":"https://github.com/icewing1996/bert-syntax","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"assessing-the-ability-of-lstms-to-learn","repo_url":"https://github.com/ketranm/fan_vs_rnn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"assessing-the-ability-of-lstms-to-learn","repo_url":"https://github.com/yoavg/bert-syntax","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1611.01368","atlas_url":"https://app.syntology.ai/?focus=1611.01368","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}