{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/recurrent-highway-networks","title":"Recurrent Highway Networks","arxiv_id":"1607.03474","date":"2016-07-12","proceeding":"ICML 2017 8","authors":["Julian Georg Zilly","Rupesh Kumar Srivastava","Jan Koutník","Jürgen Schmidhuber"],"abstract":"Many sequential processing tasks require complex nonlinear transition\nfunctions from one step to the next. However, recurrent neural networks with\n'deep' transition functions remain difficult to train, even when using Long\nShort-Term Memory (LSTM) networks. We introduce a novel theoretical analysis of\nrecurrent networks based on Gersgorin's circle theorem that illuminates several\nmodeling and optimization issues and improves our understanding of the LSTM\ncell. Based on this analysis we propose Recurrent Highway Networks, which\nextend the LSTM architecture to allow step-to-step transition depths larger\nthan one. Several language modeling experiments demonstrate that the proposed\narchitecture results in powerful and efficient models. On the Penn Treebank\ncorpus, solely increasing the transition depth from 1 to 10 improves word-level\nperplexity from 90.6 to 65.4 using the same number of parameters. On the larger\nWikipedia datasets for character prediction (text8 and enwik8), RHNs outperform\nall previous results and achieve an entropy of 1.27 bits per character.","url_abs":"http://arxiv.org/abs/1607.03474v5","url_pdf":"http://arxiv.org/pdf/1607.03474v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"recurrent-highway-networks","repo_url":"https://github.com/julian121266/RecurrentHighwayNetworks","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"recurrent-highway-networks","repo_url":"https://github.com/davidsvaughn/dts-tf","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"recurrent-highway-networks","repo_url":"https://github.com/jzilly/RecurrentHighwayNetworks","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"recurrent-highway-networks","repo_url":"https://github.com/vermaMachineLearning/Pytorch-JIT-Recurrent-Highway-Network","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"recurrent-highway-networks","repo_url":"https://github.com/labmlai/annotated_deep_learning_paper_implementations","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"recurrent-highway-networks","repo_url":"https://github.com/nanzhaogang/contrib/tree/master/application/recurrent-highway-network","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/language-modelling-on-hutter-prize","task":"Language Modelling","dataset":"Hutter Prize","model":"Large RHN","rank_in_archive_order":16,"of":18,"metrics":{"Bit per Character (BPC)":"1.27","Number of params":"46M"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-hutter-prize","task":"Language Modelling","dataset":"Hutter Prize","model":"RHN - depth 5 [zilly2016recurrent]","rank_in_archive_order":18,"of":18,"metrics":{"Bit per Character (BPC)":"1.31"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-penn-treebank-word","task":"Language Modelling","dataset":"Penn Treebank (Word Level)","model":"Recurrent highway networks","rank_in_archive_order":33,"of":43,"metrics":{"Params":"23M","Test perplexity":"65.4","Validation perplexity":"67.9"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-text8","task":"Language Modelling","dataset":"Text8","model":"Large RHN","rank_in_archive_order":17,"of":24,"metrics":{"Bit per Character (BPC)":"1.27","Number of params":"46M"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-enwiki8","task":"Language Modelling","dataset":"enwik8","model":"Recurrent Highway Networks","rank_in_archive_order":36,"of":42,"metrics":{"Bit per Character (BPC)":"1.27","Number of params":"46M"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1607.03474","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1607.03474"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nanzhaogang/contrib/tree/master/application/recurrent-highway-network","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jzilly/RecurrentHighwayNetworks","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/julian121266/RecurrentHighwayNetworks","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/davidsvaughn/dts-tf","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/labmlai/annotated_deep_learning_paper_implementations","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vermaMachineLearning/Pytorch-JIT-Recurrent-Highway-Network","reach":{"status":"ok"}}],"summary":{"ran_fixture":1,"unverified":4},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1},"listed":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"707c42ff0294c173","entry":"get_noise","repo":"jzilly/RecurrentHighwayNetworks","repo_kind":"listed","path":"rhn_train.py","file_url":"https://github.com/jzilly/RecurrentHighwayNetworks/blob/HEAD/rhn_train.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"707c42ff0294c173"}},{"code_sha256_prefix":"1830fedb612f3096","entry":"get_data","repo":"jzilly/RecurrentHighwayNetworks","repo_kind":"listed","path":"rhn_train.py","file_url":"https://github.com/jzilly/RecurrentHighwayNetworks/blob/HEAD/rhn_train.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1830fedb612f3096"}},{"code_sha256_prefix":"6a69d00503759095","entry":"hutter_raw_data","repo":"julian121266/RecurrentHighwayNetworks","repo_kind":"official","path":"theano_data.py","file_url":"https://github.com/julian121266/RecurrentHighwayNetworks/blob/HEAD/theano_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6a69d00503759095"}},{"code_sha256_prefix":"8d9ff9ce8321b114","entry":"linear","repo":"julian121266/RecurrentHighwayNetworks","repo_kind":"official","path":"rhn.py","file_url":"https://github.com/julian121266/RecurrentHighwayNetworks/blob/HEAD/rhn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8d9ff9ce8321b114"}},{"code_sha256_prefix":"65745cdcd83aced6","entry":"ptb_raw_data","repo":"julian121266/RecurrentHighwayNetworks","repo_kind":"official","path":"theano_data.py","file_url":"https://github.com/julian121266/RecurrentHighwayNetworks/blob/HEAD/theano_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"65745cdcd83aced6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}