{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/highway-networks","title":"Highway Networks","arxiv_id":"1505.00387","date":"2015-05-03","proceeding":null,"authors":["Rupesh Kumar Srivastava","Klaus Greff","Jürgen Schmidhuber"],"abstract":"There is plenty of theoretical and empirical evidence that depth of neural\nnetworks is a crucial ingredient for their success. However, network training\nbecomes more difficult with increasing depth and training of very deep networks\nremains an open problem. In this extended abstract, we introduce a new\narchitecture designed to ease gradient-based training of very deep networks. We\nrefer to networks with this architecture as highway networks, since they allow\nunimpeded information flow across several layers on \"information highways\". The\narchitecture is characterized by the use of gating units which learn to\nregulate the flow of information through a network. Highway networks with\nhundreds of layers can be trained directly using stochastic gradient descent\nand with a variety of activation functions, opening up the possibility of\nstudying extremely deep and efficient architectures.","url_abs":"http://arxiv.org/abs/1505.00387v2","url_pdf":"http://arxiv.org/pdf/1505.00387v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"highway-networks","repo_url":"https://github.com/Avmb/lowrank-highwaynetwork","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"highway-networks","repo_url":"https://github.com/SNUDerek/multiLSTM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"highway-networks","repo_url":"https://github.com/c0nn3r/pytorch_highway_networks","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"highway-networks","repo_url":"https://github.com/flukeskywalker/highway-networks","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"highway-layer","method_name":"Highway Layer"},{"method_slug":"highway-network","method_name":"Highway Network"},{"method_slug":"highway-networks","method_name":"Highway networks"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"sgd-with-momentum","method_name":"SGD with Momentum"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"}],"datasets_introduced":[],"methods_introduced":[{"slug":"highway-network","name":"Highway Network","full_name":"Highway Network"},{"slug":"highway-networks","name":"Highway networks","full_name":"Highway networks"}],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1505.00387","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}