{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/residual-lstm-design-of-a-deep-recurrent","title":"Residual LSTM: Design of a Deep Recurrent Architecture for Distant Speech Recognition","arxiv_id":"1701.03360","date":"2017-01-10","proceeding":null,"authors":["Jaeyoung Kim","Mostafa El-Khamy","Jungwon Lee"],"abstract":"In this paper, a novel architecture for a deep recurrent neural network,\nresidual LSTM is introduced. A plain LSTM has an internal memory cell that can\nlearn long term dependencies of sequential data. It also provides a temporal\nshortcut path to avoid vanishing or exploding gradients in the temporal domain.\nThe residual LSTM provides an additional spatial shortcut path from lower\nlayers for efficient training of deep networks with multiple LSTM layers.\nCompared with the previous work, highway LSTM, residual LSTM separates a\nspatial shortcut path with temporal one by using output layers, which can help\nto avoid a conflict between spatial and temporal-domain gradient flows.\nFurthermore, residual LSTM reuses the output projection matrix and the output\ngate of LSTM to control the spatial information flow instead of additional gate\nnetworks, which effectively reduces more than 10% of network parameters. An\nexperiment for distant speech recognition on the AMI SDM corpus shows that\n10-layer plain and highway LSTM networks presented 13.7% and 6.2% increase in\nWER over 3-layer aselines, respectively. On the contrary, 10-layer residual\nLSTM networks provided the lowest WER 41.0%, which corresponds to 3.3% and 2.8%\nWER reduction over plain and highway LSTM networks, respectively.","url_abs":"http://arxiv.org/abs/1701.03360v3","url_pdf":"http://arxiv.org/pdf/1701.03360v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"residual-lstm-design-of-a-deep-recurrent","repo_url":"https://github.com/kdgutier/esrnn_torch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"residual-lstm-design-of-a-deep-recurrent","repo_url":"https://github.com/tommytracey/aind-capstone","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"residual-lstm-design-of-a-deep-recurrent","repo_url":"https://github.com/kdgutier/residual_lstm/blob/master/residual_lstm.py","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"distant-speech-recognition","task_name":"Distant Speech Recognition"},{"task_slug":"speech-recognition","task_name":"Speech Recognition"},{"task_slug":"speech-recognition-1","task_name":"speech-recognition"}],"methods":[{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1701.03360","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}