{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/recurrent-additive-networks","title":"Recurrent Additive Networks","arxiv_id":"1705.07393","date":"2017-05-21","proceeding":null,"authors":["Kenton Lee","Omer Levy","Luke Zettlemoyer"],"abstract":"We introduce recurrent additive networks (RANs), a new gated RNN which is\ndistinguished by the use of purely additive latent state updates. At every time\nstep, the new state is computed as a gated component-wise sum of the input and\nthe previous state, without any of the non-linearities commonly used in RNN\ntransition dynamics. We formally show that RAN states are weighted sums of the\ninput vectors, and that the gates only contribute to computing the weights of\nthese sums. Despite this relatively simple functional form, experiments\ndemonstrate that RANs perform on par with LSTMs on benchmark language modeling\nproblems. This result shows that many of the non-linear computations in LSTMs\nand related networks are not essential, at least for the problems we consider,\nand suggests that the gates are doing more of the computational work than\npreviously understood.","url_abs":"http://arxiv.org/abs/1705.07393v2","url_pdf":"http://arxiv.org/pdf/1705.07393v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"recurrent-additive-networks","repo_url":"https://github.com/kentonl/ran","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"recurrent-additive-networks","repo_url":"https://github.com/bzhangGo/lrn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}