{"url":"/method/highway-networks","slug":"highway-networks","name":"Highway networks","full_name":"Highway networks","full_name_withheld":false,"description_markdown":"There is plenty of theoretical and empirical evidence that depth of neural networks is a crucial ingredient for their success. However, network training becomes more difficult with increasing depth and training of very deep networks remains an open problem. In this extended abstract, we introduce a new architecture designed to ease gradient-based training of very deep networks. We refer to networks with this architecture as highway networks, since they allow unimpeded information flow across several layers on \"information highways\". The architecture is characterized by the use of gating units which learn to regulate the flow of information through a network. Highway networks with hundreds of layers can be trained directly using stochastic gradient descent and with a variety of activation functions, opening up the possibility of studying extremely deep and efficient architectures.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Highway Networks","paper":"/paper/highway-networks","first_author":"Rupesh Kumar Srivastava","n_authors":3,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/highway-networks"},"source":{"url":"http://arxiv.org/abs/1505.00387v2","title":"Highway Networks","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Attention Mechanisms","url":"/methods/category/attention-mechanisms","pwc_aliases":["attention-mechanisms-1"]}],"n_papers_tagged":24,"archive_num_papers":24,"papers_newest_first":[{"paper":null,"title":"To Stay or to Bypass: Unraveling Mainline Vehicles' Aggregate Strategic Decision-Making at Highway Weaving Ramps","date":"2025-05-13","arxiv_id":"2505.08965","n_code_links":0,"syntology":null},{"paper":"/paper/gated-attention-for-large-language-models-non","title":"Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free","date":"2025-05-10","arxiv_id":"2505.06708","n_code_links":1,"syntology":{"ran":3,"of":3,"unverified":0,"pointer_only":0}},{"paper":null,"title":"Deconstructing Recurrence, Attention, and Gating: Investigating the transferability of Transformers and Gated Recurrent Neural Networks in forecasting of dynamical systems","date":"2024-10-03","arxiv_id":"2410.02654","n_code_links":0,"syntology":null},{"paper":"/paper/highway-networks-for-improved-surface","title":"Highway Networks for Improved Surface Reconstruction: The Role of Residuals and Weight Updates","date":"2024-07-11","arxiv_id":"2407.08134","n_code_links":1,"syntology":null},{"paper":null,"title":"Computer vision-based model for detecting turning lane features on Florida's public roadways","date":"2024-06-13","arxiv_id":"2406.08822","n_code_links":0,"syntology":null},{"paper":null,"title":"Deep Learning-Based Vehicle Speed Prediction for Ecological Adaptive Cruise Control in Urban and Highway Scenarios","date":"2022-11-30","arxiv_id":"2212.00149","n_code_links":0,"syntology":null},{"paper":null,"title":"Multi-task recommendation system for scientific papers with high-way networks","date":"2022-04-21","arxiv_id":"2204.09930","n_code_links":0,"syntology":null},{"paper":"/paper/transfer-learning-with-graph-neural-networks","title":"Transfer Learning with Graph Neural Networks for Short-Term Highway Traffic Forecasting","date":"2020-04-17","arxiv_id":"2004.08038","n_code_links":2,"syntology":null},{"paper":"/paper/graph-partitioning-based-diffusion","title":"Graph-Partitioning-Based Diffusion Convolutional Recurrent Neural Network for Large-Scale Traffic Forecasting","date":"2019-09-24","arxiv_id":"1909.11197","n_code_links":2,"syntology":null},{"paper":"/paper/investigating-the-effect-of-residual-and","title":"Investigating the effect of residual and highway connections in speech enhancement models","date":"2018-10-22","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":"/paper/batch-normalized-recurrent-highway-networks","title":"Batch-normalized Recurrent Highway Networks","date":"2018-09-26","arxiv_id":"1809.10271","n_code_links":1,"syntology":null},{"paper":null,"title":"Semi-tied Units for Efficient Gating in LSTM and Highway Networks","date":"2018-06-18","arxiv_id":"1806.06513","n_code_links":0,"syntology":null},{"paper":null,"title":"Humor Recognition Using Deep Learning","date":"2018-06-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"title":"Avoiding degradation in deep feed-forward networks by phasing out skip-connections","date":"2018-01-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"title":"Language Modeling with Recurrent Highway Hypernetworks","date":"2017-12-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"title":"Exploiting Nontrivial Connectivity for Automatic Speech Recognition","date":"2017-11-28","arxiv_id":"1711.10271","n_code_links":0,"syntology":null},{"paper":"/paper/lattice-recurrent-unit-improving-convergence","title":"Lattice Recurrent Unit: Improving Convergence and Statistical Efficiency for Sequence Modeling","date":"2017-10-06","arxiv_id":"1710.02254","n_code_links":3,"syntology":null},{"paper":"/paper/application-of-a-hybrid-bi-lstm-crf-model-to","title":"Application of a Hybrid Bi-LSTM-CRF model to the task of Russian Named Entity Recognition","date":"2017-09-27","arxiv_id":"1709.09686","n_code_links":2,"syntology":null},{"paper":null,"title":"Language Modeling with Highway LSTM","date":"2017-09-19","arxiv_id":"1709.06436","n_code_links":0,"syntology":null},{"paper":null,"title":"Early Improving Recurrent Elastic Highway Network","date":"2017-08-14","arxiv_id":"1708.04116","n_code_links":0,"syntology":null},{"paper":null,"title":"Faster Training of Very Deep Networks Via p-Norm Gates","date":"2016-08-11","arxiv_id":"1608.03639","n_code_links":0,"syntology":null},{"paper":null,"title":"Choice by Elimination via Deep Neural Networks","date":"2016-02-17","arxiv_id":"1602.05285","n_code_links":0,"syntology":null},{"paper":"/paper/training-very-deep-networks","title":"Training Very Deep Networks","date":"2015-07-22","arxiv_id":"1507.06228","n_code_links":3,"syntology":null},{"paper":"/paper/highway-networks","title":"Highway Networks","date":"2015-05-03","arxiv_id":"1505.00387","n_code_links":4,"syntology":null}],"papers_shown":24,"tasks":[{"task":"/task/language-modelling","name":"Language Modelling","papers":4},{"task":"/task/language-modeling","name":"Language Modeling","papers":3},{"task":"/task/speech-recognition","name":"Speech Recognition","papers":3},{"task":"/task/speech-recognition-1","name":"speech-recognition","papers":3},{"task":"/task/automatic-speech-recognition-2","name":"Automatic Speech Recognition","papers":2},{"task":"/task/automatic-speech-recognition","name":"Automatic Speech Recognition (ASR)","papers":2},{"task":"/task/image-classification","name":"Image Classification","papers":2},{"task":"/task/management","name":"Management","papers":2},{"task":"/task/time-series","name":"Time Series Analysis","papers":2},{"task":"/task/activity-recognition","name":"Activity Recognition","papers":1},{"task":"/task/attribute","name":"Attribute","papers":1},{"task":"/task/decision-making","name":"Decision Making","papers":1},{"task":"/task/deep-learning","name":"Deep Learning","papers":1},{"task":"/task/denoising","name":"Denoising","papers":1},{"task":"/task/classification","name":"General Classification","papers":1},{"task":"/task/graph-neural-network","name":"Graph Neural Network","papers":1},{"task":"/task/human-activity-recognition","name":"Human Activity Recognition","papers":1},{"task":"/task/image-captioning","name":"Image Captioning","papers":1},{"task":"/task/learning-to-rank","name":"Learning-To-Rank","papers":1},{"task":"/task/machine-translation","name":"Machine Translation","papers":1}],"tasks_shown":20,"n_tasks":36,"usage_by_year":[{"year":"2015","papers":2},{"year":"2016","papers":2},{"year":"2017","papers":6},{"year":"2018","papers":5},{"year":"2019","papers":1},{"year":"2020","papers":1},{"year":"2022","papers":2},{"year":"2024","papers":3},{"year":"2025","papers":2}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/highway-networks"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}