Papers › The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation

The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation

26 Apr 2018ACL 2018 7arXiv:1804.09849archive 2025-07-28

Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Mike Schuster, Zhifeng Chen, Yonghui Wu, Macduff Hughes

The past year has witnessed rapid advances in sequence-to-sequence (seq2seq) modeling for Machine Translation (MT). The classic RNN-based approaches to MT were first out-performed by the convolutional seq2seq model, which was then out-performed by the more recent Transformer model. Each of these new approaches consists of a fundamental architecture accompanied by a set of modeling and training techniques that are in principle applicable to other seq2seq architectures. In this paper, we tease apart the new architectures and their accompanying techniques in two ways. First, we identify several key modeling and training techniques, and apply them to the RNN architecture, yielding a new RNMT+ model that outperforms all of the three fundamental architectures on the benchmark WMT'14 English to French and English to German tasks. Second, we analyze the properties of each fundamental seq2seq architecture and devise new hybrid architectures intended to combine their strengths. Our hybrid models obtain further improvements, outperforming the RNMT+ model on both benchmark datasets.

PaperPDFConference PDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

tensorflow/lingvo officialtfApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to N-Grammer: Augmenting Transformers with latent n-grams (arXiv:2207.06366; that paper's own run record: 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
duyvuleo/Transformer-DyNet mentioned on GitHubtfMITSyntology: not harvested report
zysite/post mentioned on GitHubpytorchMITSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Machine TranslationTranslation

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Machine Translation WMT2014 English-French RNMT+ BLEU score 41.0 #27 of 57 Archive leaderboard report
Machine Translation WMT2014 English-French RNMT+ Hardware Burden 132G #27 of 57 Archive leaderboard report
Machine Translation WMT2014 English-French RNMT+ Operations per network pass 2.81G #27 of 57 Archive leaderboard report
Machine Translation WMT2014 English-German RNMT+ BLEU score 28.5 #41 of 91 Archive leaderboard report
Machine Translation WMT2014 English-German RNMT+ Hardware Burden 44G #41 of 91 Archive leaderboard report
Machine Translation WMT2014 English-German RNMT+ Operations per network pass 2.81G #41 of 91 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Absolute Position EncodingsAdamAttentionBPEDense ConnectionsDropoutLSTMLabel SmoothingLayer NormalizationLinear LayerMulti-Head AttentionPosition-Wise Feed-Forward LayerReLUResidual ConnectionSeq2SeqSigmoid ActivationSoftmaxTanh ActivationTransformer

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections