{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/neural-machine-translation-in-linear-time","title":"Neural Machine Translation in Linear Time","arxiv_id":"1610.10099","date":"2016-10-31","proceeding":null,"authors":["Nal Kalchbrenner","Lasse Espeholt","Karen Simonyan","Aaron van den Oord","Alex Graves","Koray Kavukcuoglu"],"abstract":"We present a novel neural network for processing sequences. The ByteNet is a\none-dimensional convolutional neural network that is composed of two parts, one\nto encode the source sequence and the other to decode the target sequence. The\ntwo network parts are connected by stacking the decoder on top of the encoder\nand preserving the temporal resolution of the sequences. To address the\ndiffering lengths of the source and the target, we introduce an efficient\nmechanism by which the decoder is dynamically unfolded over the representation\nof the encoder. The ByteNet uses dilation in the convolutional layers to\nincrease its receptive field. The resulting network has two core properties: it\nruns in time that is linear in the length of the sequences and it sidesteps the\nneed for excessive memorization. The ByteNet decoder attains state-of-the-art\nperformance on character-level language modelling and outperforms the previous\nbest results obtained with recurrent networks. The ByteNet also achieves\nstate-of-the-art performance on character-to-character machine translation on\nthe English-to-German WMT translation task, surpassing comparable neural\ntranslation models that are based on recurrent networks with attentional\npooling and run in quadratic time. We find that the latent alignment structure\ncontained in the representations reflects the expected alignment between the\ntokens.","url_abs":"http://arxiv.org/abs/1610.10099v2","url_pdf":"http://arxiv.org/pdf/1610.10099v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"neural-machine-translation-in-linear-time","repo_url":"https://github.com/Shivendra-psc/speechbot","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"neural-machine-translation-in-linear-time","repo_url":"https://github.com/Vikas-Sony/speech-to-text","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"neural-machine-translation-in-linear-time","repo_url":"https://github.com/adityaagrawal7/speech-to-text-wavenet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"neural-machine-translation-in-linear-time","repo_url":"https://github.com/freedombenLiu/speech-to-text-wavenet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"neural-machine-translation-in-linear-time","repo_url":"https://github.com/kingstarcraft/speech-to-text-wavenet2","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"neural-machine-translation-in-linear-time","repo_url":"https://github.com/kinimod23/ATS_Project","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"neural-machine-translation-in-linear-time","repo_url":"https://github.com/liguigui/speech-to-text-wavenet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"neural-machine-translation-in-linear-time","repo_url":"https://github.com/microsoft/protein-sequence-models","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"neural-machine-translation-in-linear-time","repo_url":"https://github.com/paarthneekhara/byteNet-tensorflow","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"neural-machine-translation-in-linear-time","repo_url":"https://github.com/randomrandom/deep-atrous-cnn-sentiment","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"neural-machine-translation-in-linear-time","repo_url":"https://github.com/sriharireddypusapati/speech-to-text-wavenet2","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"memorization","task_name":"Memorization"},{"task_slug":"translation","task_name":"Translation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/language-modelling-on-enwiki8","task":"Language Modelling","dataset":"enwik8","model":"ByteNet","rank_in_archive_order":37,"of":42,"metrics":{"Bit per Character (BPC)":"1.31"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2014-english-german","task":"Machine Translation","dataset":"WMT2014 English-German","model":"ByteNet","rank_in_archive_order":71,"of":91,"metrics":{"BLEU score":"23.75"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2015-english-german","task":"Machine Translation","dataset":"WMT2015 English-German","model":"ByteNet","rank_in_archive_order":1,"of":6,"metrics":{"BLEU score":"26.3"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1610.10099","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}