{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/representation-of-linguistic-form-and","title":"Representation of linguistic form and function in recurrent neural networks","arxiv_id":"1602.08952","date":"2016-02-29","proceeding":"CL 2017 12","authors":["Ákos Kádár","Grzegorz Chrupała","Afra Alishahi"],"abstract":"We present novel methods for analyzing the activation patterns of RNNs from a\nlinguistic point of view and explore the types of linguistic structure they\nlearn. As a case study, we use a multi-task gated recurrent network\narchitecture consisting of two parallel pathways with shared word embeddings\ntrained on predicting the representations of the visual scene corresponding to\nan input sentence, and predicting the next word in the same sentence. Based on\nour proposed method to estimate the amount of contribution of individual tokens\nin the input to the final prediction of the networks we show that the image\nprediction pathway: a) is sensitive to the information structure of the\nsentence b) pays selective attention to lexical categories and grammatical\nfunctions that carry semantic information c) learns to treat the same input\ntoken differently depending on its grammatical functions in the sentence. In\ncontrast the language model is comparatively more sensitive to words with a\nsyntactic function. Furthermore, we propose methods to ex- plore the function\nof individual hidden units in RNNs and show that the two pathways of the\narchitecture in our case study contain specialized units tuned to patterns\ninformative for the task, some of which can carry activations to later time\nsteps to encode long-term dependencies.","url_abs":"http://arxiv.org/abs/1602.08952v2","url_pdf":"http://arxiv.org/pdf/1602.08952v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"representation-of-linguistic-form-and","repo_url":"https://github.com/phnk/D7047E","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"form","task_name":"Form"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"word-embeddings","task_name":"Word Embeddings"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1602.08952","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}