{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-comparative-study-on-transformer-vs-rnn-in","title":"A Comparative Study on Transformer vs RNN in Speech Applications","arxiv_id":"1909.06317","date":"2019-09-13","proceeding":null,"authors":["Shigeki Karita","Nanxin Chen","Tomoki Hayashi","Takaaki Hori","Hirofumi Inaguma","Ziyan Jiang","Masao Someki","Nelson Enrique Yalta Soplin","Ryuichi Yamamoto","Xiaofei Wang","Shinji Watanabe","Takenori Yoshimura","Wangyou Zhang"],"abstract":"Sequence-to-sequence models have been widely used in end-to-end speech processing, for example, automatic speech recognition (ASR), speech translation (ST), and text-to-speech (TTS). This paper focuses on an emergent sequence-to-sequence model called Transformer, which achieves state-of-the-art performance in neural machine translation and other natural language processing applications. We undertook intensive studies in which we experimentally compared and analyzed Transformer and conventional recurrent neural networks (RNN) in a total of 15 ASR, one multilingual ASR, one ST, and two TTS benchmarks. Our experiments revealed various training tips and significant performance benefits obtained with Transformer for each task including the surprising superiority of Transformer in 13/15 ASR benchmarks in comparison with RNN. We are preparing to release Kaldi-style reproducible recipes using open source and publicly available datasets for all the ASR, ST, and TTS tasks for the community to succeed our exciting outcomes.","url_abs":"https://arxiv.org/abs/1909.06317v2","url_pdf":"https://arxiv.org/pdf/1909.06317v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-comparative-study-on-transformer-vs-rnn-in","repo_url":"https://github.com/espnet/espnet","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"a-comparative-study-on-transformer-vs-rnn-in","repo_url":"https://github.com/MindSpore-scientific-2/code-11/tree/main/ZoneoutRNN","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"automatic-speech-recognition-2","task_name":"Automatic Speech Recognition"},{"task_slug":"automatic-speech-recognition","task_name":"Automatic Speech Recognition (ASR)"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"speech-recognition","task_name":"Speech Recognition"},{"task_slug":"text-to-speech","task_name":"Text to Speech"},{"task_slug":"translation","task_name":"Translation"},{"task_slug":"speech-recognition-1","task_name":"speech-recognition"},{"task_slug":"text-to-speech-1","task_name":"text-to-speech"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/speech-recognition-on-aishell-1","task":"Speech Recognition","dataset":"AISHELL-1","model":"CTC/Att","rank_in_archive_order":17,"of":18,"metrics":{"Word Error Rate (WER)":"6.7"},"uses_additional_data":false},{"leaderboard":"/sota/speech-recognition-on-librispeech-test-clean","task":"Speech Recognition","dataset":"LibriSpeech test-clean","model":"Transformer","rank_in_archive_order":40,"of":64,"metrics":{"Word Error Rate (WER)":"2.6"},"uses_additional_data":true},{"leaderboard":"/sota/speech-recognition-on-librispeech-test-other","task":"Speech Recognition","dataset":"LibriSpeech test-other","model":"Transformer","rank_in_archive_order":35,"of":53,"metrics":{"Word Error Rate (WER)":"5.7"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1909.06317","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}