{"url":"/method/highway-layer","slug":"highway-layer","name":"Highway Layer","full_name":"Highway Layer","full_name_withheld":false,"description_markdown":"A **Highway Layer** contains an information highway to other layers that helps with information flow. It is characterised by the use of a gating unit to help this information flow. \r\n\r\nA plain feedforward neural network typically consists of $L$ layers where the $l$th layer ($l \\in ${$1, 2, \\dots, L$}) applies a nonlinear transform $H$ (parameterized by $\\mathbf{W\\_{H,l}}$) on its input $\\mathbf{x\\_{l}}$ to produce its output $\\mathbf{y\\_{l}}$. Thus, $\\mathbf{x\\_{1}}$ is the input to the network and $\\mathbf{y\\_{L}}$ is the network’s output. Omitting the layer index and biases for clarity,\r\n\r\n$$ \\mathbf{y} = H\\left(\\mathbf{x},\\mathbf{W\\_{H}}\\right) $$\r\n\r\n$H$ is usually an affine transform followed by a non-linear activation function, but in general it may take other forms. \r\n\r\nFor a [highway network](https://paperswithcode.com/method/highway-network), we additionally define two nonlinear transforms $T\\left(\\mathbf{x},\\mathbf{W\\_{T}}\\right)$ and $C\\left(\\mathbf{x},\\mathbf{W\\_{C}}\\right)$ such that:\r\n\r\n$$ \\mathbf{y} = H\\left(\\mathbf{x},\\mathbf{W\\_{H}}\\right)·T\\left(\\mathbf{x},\\mathbf{W\\_{T}}\\right) + \\mathbf{x}·C\\left(\\mathbf{x},\\mathbf{W\\_{C}}\\right)$$\r\n\r\nWe refer to T as the transform gate and C as the carry gate, since they express how much of the output is produced by transforming the input and carrying it, respectively. In the original paper, the authors set $C = 1 − T$, giving:\r\n\r\n$$ \\mathbf{y} = H\\left(\\mathbf{x},\\mathbf{W\\_{H}}\\right)·T\\left(\\mathbf{x},\\mathbf{W\\_{T}}\\right) + \\mathbf{x}·\\left(1-T\\left(\\mathbf{x},\\mathbf{W\\_{T}}\\right)\\right)$$\r\n\r\nThe authors set:\r\n\r\n$$ T\\left(x\\right) = \\sigma\\left(\\mathbf{W\\_{T}}^{T}\\mathbf{x} + \\mathbf{b\\_{T}}\\right) $$\r\n\r\nImage: [Sik-Ho Tsang](https://towardsdatascience.com/review-highway-networks-gating-function-to-highway-image-classification-5a33833797b5)","description_state":"present","introduced_year":null,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":"http://arxiv.org/abs/1505.00387v2","title":"Highway Networks","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/kefirski/pytorch_Highway/blob/70b75db7a2d029f4bbe08fd4c7d69e36bf7b6d3a/highway/highway.py#L35","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Miscellaneous Components","url":"/methods/category/miscellaneous-components","pwc_aliases":[]}],"n_papers_tagged":108,"archive_num_papers":null,"papers_newest_first":[{"paper":null,"title":"Coordinated Ramp Metering Control based on Scalable Nonlinear Traffic Dynamics Model Discovery in a Large Network","date":"2025-03-09","arxiv_id":"2503.06767","n_code_links":0,"syntology":null},{"paper":"/paper/very-attentive-tacotron-robust-and-unbounded","title":"Robust and Unbounded Length Generalization in Autoregressive Transformer-Based Text-to-Speech","date":"2024-10-29","arxiv_id":"2410.22179","n_code_links":1,"syntology":null},{"paper":null,"title":"Enhancing Kurdish Text-to-Speech with Native Corpus Training: A High-Quality WaveGlow Vocoder Approach","date":"2024-09-10","arxiv_id":"2409.13734","n_code_links":0,"syntology":null},{"paper":null,"title":"Training Universal Vocoders with Feature Smoothing-Based Augmentation Methods for High-Quality TTS Systems","date":"2024-09-04","arxiv_id":"2409.02517","n_code_links":0,"syntology":null},{"paper":"/paper/expanding-the-scope-inductive-knowledge-graph","title":"Expanding the Scope: Inductive Knowledge Graph Reasoning with Multi-Starting Progressive Propagation","date":"2024-07-15","arxiv_id":"2407.10430","n_code_links":1,"syntology":null},{"paper":"/paper/highway-networks-for-improved-surface","title":"Highway Networks for Improved Surface Reconstruction: The Role of Residuals and Weight Updates","date":"2024-07-11","arxiv_id":"2407.08134","n_code_links":1,"syntology":null},{"paper":null,"title":"Stable Weight Updating: A Key to Reliable PDE Solutions Using Deep Learning","date":"2024-07-10","arxiv_id":"2407.07375","n_code_links":0,"syntology":null},{"paper":"/paper/modeling-of-spatially-embedded-networks-via","title":"Modeling of spatially embedded networks via regional spatial graph convolutional networks","date":"2024-06-20","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":null,"title":"Informed along the road: roadway capacity driven graph convolution network for network-wide traffic prediction","date":"2024-06-18","arxiv_id":"2406.13057","n_code_links":0,"syntology":null},{"paper":null,"title":"Traffic Prediction considering Multiple Levels of Spatial-temporal Information: A Multi-scale Graph Wavelet-based Approach","date":"2024-06-18","arxiv_id":"2406.13038","n_code_links":0,"syntology":null},{"paper":null,"title":"Leveraging the Interplay Between Syntactic and Acoustic Cues for Optimizing Korean TTS Pause Formation","date":"2024-04-03","arxiv_id":"2404.02592","n_code_links":0,"syntology":null},{"paper":"/paper/selective-stereo-adaptive-frequency","title":"Selective-Stereo: Adaptive Frequency Information Selection for Stereo Matching","date":"2024-03-01","arxiv_id":"2403.00486","n_code_links":1,"syntology":{"ran":9,"of":13,"unverified":4,"pointer_only":0}},{"paper":"/paper/on-the-effectiveness-of-unlearning-in-session","title":"On the Effectiveness of Unlearning in Session-Based Recommendation","date":"2023-12-22","arxiv_id":"2312.14447","n_code_links":1,"syntology":{"ran":1,"of":1,"unverified":0,"pointer_only":1}},{"paper":null,"title":"A Data-Driven Integrated Framework for Fast-Charging Facility Planning using Multi-Period Bi-Objective Optimization","date":"2023-11-21","arxiv_id":"2311.12700","n_code_links":0,"syntology":null},{"paper":"/paper/reinforcement-learning-with-model-predictive","title":"Reinforcement Learning with Model Predictive Control for Highway Ramp Metering","date":"2023-11-15","arxiv_id":"2311.08820","n_code_links":1,"syntology":null},{"paper":"/paper/distil-the-informative-essence-of-loop","title":"Distil the informative essence of loop detector data set: Is network-level traffic forecasting hungry for more data?","date":"2023-10-31","arxiv_id":"2310.20366","n_code_links":1,"syntology":null},{"paper":"/paper/physics-informed-with-power-enhanced-residual","title":"Power-Enhanced Residual Network for Function Approximation and Physics-Informed Inverse Problems","date":"2023-10-24","arxiv_id":"2310.15690","n_code_links":1,"syntology":null},{"paper":null,"title":"An overview of text-to-speech systems and media applications","date":"2023-10-22","arxiv_id":"2310.14301","n_code_links":0,"syntology":null},{"paper":null,"title":"Energy-Based Models For Speech Synthesis","date":"2023-10-19","arxiv_id":"2310.12765","n_code_links":0,"syntology":null},{"paper":null,"title":"Facilitating Battery Swapping Services for Freight Trucks with Spatial-Temporal Demand Prediction","date":"2023-10-01","arxiv_id":"2310.04440","n_code_links":0,"syntology":null},{"paper":null,"title":"The DeepZen Speech Synthesis System for Blizzard Challenge 2023","date":"2023-08-30","arxiv_id":"2308.15945","n_code_links":0,"syntology":null},{"paper":null,"title":"Machine learning for option pricing: an empirical investigation of network architectures","date":"2023-07-14","arxiv_id":"2307.07657","n_code_links":0,"syntology":null},{"paper":"/paper/multilingual-text-to-speech-synthesis-for","title":"Multilingual Text-to-Speech Synthesis for Turkic Languages Using Transliteration","date":"2023-05-25","arxiv_id":"2305.15749","n_code_links":1,"syntology":null},{"paper":"/paper/neural-machine-translation-models-with","title":"Neural Machine Translation Models with Attention-Based Dropout Layer","date":"2023-05-01","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":null,"title":"A Virtual Simulation-Pilot Agent for Training of Air Traffic Controllers","date":"2023-04-16","arxiv_id":"2304.07842","n_code_links":0,"syntology":null},{"paper":null,"title":"ArmanTTS single-speaker Persian dataset","date":"2023-04-07","arxiv_id":"2304.03585","n_code_links":0,"syntology":null},{"paper":"/paper/scconv-spatial-and-channel-reconstruction","title":"SCConv: Spatial and Channel Reconstruction Convolution for Feature Redundancy","date":"2023-01-01","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":null,"title":"Investigation of Japanese PnG BERT language model in text-to-speech synthesis for pitch accent language","date":"2022-12-16","arxiv_id":"2212.08321","n_code_links":0,"syntology":null},{"paper":null,"title":"Investigating Content-Aware Neural Text-To-Speech MOS Prediction Using Prosodic and Linguistic Features","date":"2022-11-01","arxiv_id":"2211.00342","n_code_links":0,"syntology":null},{"paper":null,"title":"Cross-lingual Text-To-Speech with Flow-based Voice Conversion for Improved Pronunciation","date":"2022-10-31","arxiv_id":"2210.17264","n_code_links":0,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/speech-synthesis","name":"Speech Synthesis","papers":44},{"task":"/task/text-to-speech","name":"Text to Speech","papers":41},{"task":"/task/text-to-speech-1","name":"text-to-speech","papers":41},{"task":"/task/text-to-speech-synthesis","name":"Text-To-Speech Synthesis","papers":15},{"task":"/task/decoder","name":"Decoder","papers":11},{"task":"/task/speech-recognition","name":"Speech Recognition","papers":9},{"task":"/task/language-modelling","name":"Language Modelling","papers":8},{"task":"/task/speech-recognition-1","name":"speech-recognition","papers":8},{"task":"/task/language-modeling","name":"Language Modeling","papers":6},{"task":"/task/sentence","name":"Sentence","papers":6},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":5},{"task":"/task/voice-cloning","name":"Voice Cloning","papers":5},{"task":"/task/data-augmentation","name":"Data Augmentation","papers":4},{"task":"/task/voice-conversion","name":"Voice Conversion","papers":4},{"task":"/task/audio-synthesis","name":"Audio Synthesis","papers":3},{"task":null,"name":"CPU","papers":3},{"task":"/task/diversity","name":"Diversity","papers":3},{"task":"/task/expressive-speech-synthesis","name":"Expressive Speech Synthesis","papers":3},{"task":null,"name":"GPU","papers":3},{"task":"/task/classification","name":"General Classification","papers":3}],"tasks_shown":20,"n_tasks":96,"usage_by_year":[{"year":"2015","papers":3},{"year":"2016","papers":1},{"year":"2017","papers":6},{"year":"2018","papers":15},{"year":"2019","papers":11},{"year":"2020","papers":21},{"year":"2021","papers":14},{"year":"2022","papers":10},{"year":"2023","papers":15},{"year":"2024","papers":11},{"year":"2025","papers":1}],"row_source":"embedded","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/highway-layer"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}