{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/recurrent-orthogonal-networks-and-long-memory","title":"Recurrent Orthogonal Networks and Long-Memory Tasks","arxiv_id":"1602.06662","date":"2016-02-22","proceeding":null,"authors":["Mikael Henaff","Arthur Szlam","Yann Lecun"],"abstract":"Although RNNs have been shown to be powerful tools for processing sequential\ndata, finding architectures or optimization strategies that allow them to model\nvery long term dependencies is still an active area of research. In this work,\nwe carefully analyze two synthetic datasets originally outlined in (Hochreiter\nand Schmidhuber, 1997) which are used to evaluate the ability of RNNs to store\ninformation over many time steps. We explicitly construct RNN solutions to\nthese problems, and using these constructions, illuminate both the problems\nthemselves and the way in which RNNs store different types of information in\ntheir hidden states. These constructions furthermore explain the success of\nrecent methods that specify unitary initializations or constraints on the\ntransition matrices.","url_abs":"http://arxiv.org/abs/1602.06662v2","url_pdf":"http://arxiv.org/pdf/1602.06662v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"recurrent-orthogonal-networks-and-long-memory","repo_url":"https://github.com/solgaardlab/neurophox","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1602.06662","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}