{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/light-gated-recurrent-units-for-speech","title":"Light Gated Recurrent Units for Speech Recognition","arxiv_id":"1803.10225","date":"2018-03-26","proceeding":null,"authors":["Mirco Ravanelli","Philemon Brakel","Maurizio Omologo","Yoshua Bengio"],"abstract":"A field that has directly benefited from the recent advances in deep learning\nis Automatic Speech Recognition (ASR). Despite the great achievements of the\npast decades, however, a natural and robust human-machine speech interaction\nstill appears to be out of reach, especially in challenging environments\ncharacterized by significant noise and reverberation. To improve robustness,\nmodern speech recognizers often employ acoustic models based on Recurrent\nNeural Networks (RNNs), that are naturally able to exploit large time contexts\nand long-term speech modulations. It is thus of great interest to continue the\nstudy of proper techniques for improving the effectiveness of RNNs in\nprocessing speech signals.\n  In this paper, we revise one of the most popular RNN models, namely Gated\nRecurrent Units (GRUs), and propose a simplified architecture that turned out\nto be very effective for ASR. The contribution of this work is two-fold: First,\nwe analyze the role played by the reset gate, showing that a significant\nredundancy with the update gate occurs. As a result, we propose to remove the\nformer from the GRU design, leading to a more efficient and compact single-gate\nmodel. Second, we propose to replace hyperbolic tangent with ReLU activations.\nThis variation couples well with batch normalization and could help the model\nlearn long-term dependencies without numerical issues.\n  Results show that the proposed architecture, called Light GRU (Li-GRU), not\nonly reduces the per-epoch training time by more than 30% over a standard GRU,\nbut also consistently improves the recognition accuracy across different tasks,\ninput features, noisy conditions, as well as across different ASR paradigms,\nranging from standard DNN-HMM speech recognizers to end-to-end CTC models.","url_abs":"http://arxiv.org/abs/1803.10225v1","url_pdf":"http://arxiv.org/pdf/1803.10225v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"light-gated-recurrent-units-for-speech","repo_url":"https://github.com/mravanelli/theano-kaldi-rnn","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"automatic-speech-recognition-2","task_name":"Automatic Speech Recognition"},{"task_slug":"automatic-speech-recognition","task_name":"Automatic Speech Recognition (ASR)"},{"task_slug":"speech-recognition","task_name":"Speech Recognition"},{"task_slug":"speech-recognition-1","task_name":"speech-recognition"}],"methods":[{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"gru","method_name":"GRU"},{"method_slug":"relu","method_name":"ReLU"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/speech-recognition-on-timit","task":"Speech Recognition","dataset":"TIMIT","model":"Li-GRU + fMLLR features","rank_in_archive_order":7,"of":22,"metrics":{"Percentage error":"14.9"},"uses_additional_data":false},{"leaderboard":"/sota/speech-recognition-on-timit","task":"Speech Recognition","dataset":"TIMIT","model":"Light Gated Recurrent Units","rank_in_archive_order":15,"of":22,"metrics":{"Percentage error":"16.7"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1803.10225","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}