{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/efficient-orthogonal-parametrisation-of","title":"Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections","arxiv_id":"1612.00188","date":"2016-12-01","proceeding":"ICML 2017 8","authors":["Zakaria Mhammedi","Andrew Hellicar","Ashfaqur Rahman","James Bailey"],"abstract":"The problem of learning long-term dependencies in sequences using Recurrent\nNeural Networks (RNNs) is still a major challenge. Recent methods have been\nsuggested to solve this problem by constraining the transition matrix to be\nunitary during training which ensures that its norm is equal to one and\nprevents exploding gradients. These methods either have limited expressiveness\nor scale poorly with the size of the network when compared with the simple RNN\ncase, especially when using stochastic gradient descent with a small mini-batch\nsize. Our contributions are as follows; we first show that constraining the\ntransition matrix to be unitary is a special case of an orthogonal constraint.\nThen we present a new parametrisation of the transition matrix which allows\nefficient training of an RNN while ensuring that the matrix is always\northogonal. Our results show that the orthogonal constraint on the transition\nmatrix applied through our parametrisation gives similar benefits to the\nunitary constraint, without the time complexity limitations.","url_abs":"http://arxiv.org/abs/1612.00188v5","url_pdf":"http://arxiv.org/pdf/1612.00188v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"efficient-orthogonal-parametrisation-of","repo_url":"https://github.com/zmhammedi/Orthogonal_RNN","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1612.00188","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}