Datasets › WMT 2014

WMT 2014

Introduced by Ondrej Bojar et al. in Findings of the 2014 Workshop on Statistical Machine Translation1 Jan 2014 archive 2025-07-28

WMT 2014 is a collection of datasets used in shared tasks of the Ninth Workshop on Statistical Machine Translation. The workshop featured four tasks:

  • a news translation task,
  • a quality estimation task,
  • a metrics task,
  • a medical text translation task.

Source: https://www.aclweb.org/anthology/W14-3302.pdf

Benchmarks archive 2025-07-28

All 9 leaderboards whose dataset resolves to this page shown (sort by any header). "First row" is the archive's own first row at snapshot, in the archive's row order; nothing here re-ranks and metric direction is not asserted.

Syntology 1 newer paper was judged to report on this dataset's benchmarks so far (Claude Sonnet 4.5, reading their tables: a model's judgement, not a result; 0 placed): Machine Translation · WMT2014 German-English (1 judged, 0 placed), Unsupervised Machine Translation · WMT2014 German-English (1 judged, 0 placed). Drawn from 8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read).

First row (archive order)PaperCode
Machine Translation WMT2014 English-German Transformer Cycle (Rev) BLEU score 35.14 Lessons on Parameter Sharing across Layers in Transformers takase/share_layer_params +1 91 Compare
Machine Translation WMT2014 English-French Transformer+BT (ADMIN init) BLEU score 46.4 Very Deep Transformers for Neural Machine Translation LiyuanLucasLiu/Transforemr-Clinic +3 57 Compare
Machine Translation WMT2014 German-English Bi-SimCut BLEU score 35.15 Bi-SimCut: A Simple Strategy for Boosting Neural Machine... gpengzhi/Bi-SimCut 16 Compare
Unsupervised Machine Translation WMT2014 French-English GPT-3 175B (Few-Shot) BLEU 39.2 Language Models are Few-Shot Learners ggml-org/llama.cpp +66 7 Compare
Unsupervised Machine Translation WMT2014 English-French BERT-fused NMT BLEU 38.27 Incorporating BERT into Neural Machine Translation bert-nmt/bert-nmt +2 7 Compare
Machine Translation WMT2014 French-English FLAN 137B (few-shot, k=9) BLEU score 37.9 Finetuned Language Models Are Zero-Shot Learners hiyouga/llama-efficient-tuning +7 3 Compare
Machine Translation WMT2014 English-Czech Evolved Transformer Big BLEU score 28.2 The Evolved Transformer tensorflow/tensor2tensor +2 2 Compare
Unsupervised Machine Translation WMT2014 English-German SMT + NMT (tuning and joint refinement) BLEU 22.5 An Effective Approach to Unsupervised Machine Translation artetxem/monoses 2 Compare
Unsupervised Machine Translation WMT2014 German-English SMT + NMT (tuning and joint refinement) BLEU 27.0 An Effective Approach to Unsupervised Machine Translation artetxem/monoses 2 Compare

Papers archive 2025-07-28

papers with a benchmark row: 89 · with a code link: 81 · where Syntology ran a sample: 40 (33 with a run with no instrument failure, 7 where every run was a failure of Syntology's instrument) Syntology

Show: all papers with a benchmark rowonly where code ran (40 of 89 with a benchmark row: 33 with a run with no instrument failure, 7 where every run was a failure of Syntology's instrument)

Page 1 of 1: papers 1 to 89 of 89 with a leaderboard row on this dataset's benchmarks, newest first by the archive's date (ties by slug; undated papers last).

The archive's own "papers using this dataset" list was never published, so this is the benchmark-backed subset, not that list; the archive's count for this dataset is 288. The Syntology column is from Syntology's graph, stated per sample; it is not part of any archive number. A line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified” (C is a part of N, never taken away from it); the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. When the archive marks a repository official for the paper, the cell starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record.

PaperCodeResultsDateSamples run Syntology
PartialFormer: Modeling Part Instead of Whole for Machine Translation 1 1 23 Oct 2023 not harvested
Mega: Moving Average Equipped Gated Attention 7 2 21 Sep 2022 official (archive's flag): 3 ran · 12 ran (of which 6 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 5 where Syntology's instrument failed) · 1 unverified (11 pointer-only for licence)
Bi-SimCut: A Simple Strategy for Boosting Neural Machine Translation 1 4 6 Jun 2022 not harvested
BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation 2 2 9 Sep 2021 official (archive's flag): 1 ran · 2 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (1 pointer-only for licence)
Finetuned Language Models Are Zero-Shot Learners 8 4 3 Sep 2021 official: harvested for another paper · 0 ran · 1 unverified
R-Drop: Regularized Dropout for Neural Networks 8 2 28 Jun 2021 community repositories only · 4 ran (of which 2 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified
ResMLP: Feedforward networks for image classification with data-efficient training 19 4 7 May 2021 community repositories only · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 1 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified
Lessons on Parameter Sharing across Layers in Transformers 2 1 13 Apr 2021 official (archive's flag): 2 ran · 2 ran (of which 2 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified; every one of the 2 samples that ran constructed an object rather than computing a result (2 pointer-only for licence)
Rethinking Perturbations in Encoder-Decoders for Fast Training 1 1 5 Apr 2021 official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (1 pointer-only for licence)
Mask Attention Networks: Rethinking and Strengthen Transformer 1 2 25 Mar 2021 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified
Finetuning Pretrained Transformers into RNNs 2 2 24 Mar 2021 3 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 5 unverified (4 pointer-only for licence)
Non-Autoregressive Translation by Learning Target Categorical Codes 1 2 21 Mar 2021 not harvested
Random Feature Attention 0 2 3 Mar 2021 not harvested
OmniNet: Omnidirectional Representations from Transformers 1 2 1 Mar 2021 not harvested
AutoDropout: Learning Dropout Patterns to Regularize Deep Networks 1 1 5 Jan 2021 not harvested
Subformer: A Parameter Reduced Transformer 0 1 1 Jan 2021 not harvested
Incorporating a Local Translation Mechanism into Non-autoregressive Translation 1 4 12 Nov 2020 official (archive's flag): 3 ran · 3 ran (of which 2 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified
Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information 1 1 7 Oct 2020 not harvested
Glancing Transformer for Non-Autoregressive Neural Machine Translation 2 1 18 Aug 2020 not harvested
Very Deep Transformers for Neural Machine Translation 4 3 18 Aug 2020 community repositories only · 8 ran (of which 2 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 5 where Syntology's instrument failed) · 1 unverified (3 pointer-only for licence)
AdvAug: Robust Adversarial Augmentation for Neural Machine Translation 0 3 21 Jun 2020 not harvested
Multi-branch Attentive Transformer 1 1 18 Jun 2020 3 ran (of which 2 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (3 pointer-only for licence)
HAT: Hardware-Aware Transformers for Efficient Natural Language Processing 4 2 28 May 2020 not harvested
Language Models are Few-Shot Learners 67 2 28 May 2020 community repositories only · 45 ran (of which 0 constructed an object rather than computing a result; 37 with no instrument failure: 3 honoured, 2 violated, 32 with no contract checked; 8 where Syntology's instrument failed) · 20 unverified (7 pointer-only for licence)
Synthesizer: Rethinking Self-Attention in Transformer Models 1 2 2 May 2020 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result
Lite Transformer with Long-Short Range Attention 2 2 24 Apr 2020 not harvested
Understanding the Difficulty of Training Transformers 2 1 17 Apr 2020 official (archive's flag): 1 ran · 4 ran (of which 2 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified
PowerNorm: Rethinking Batch Normalization in Transformers 1 1 17 Mar 2020 not harvested
Learning to Encode Position for Transformer with Continuous Dynamical Model 1 2 13 Mar 2020 3 ran (of which 2 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 3 unverified (6 pointer-only for licence)
Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule 5 1 9 Mar 2020 not harvested
Incorporating BERT into Neural Machine Translation 3 3 17 Feb 2020 not harvested
Time-aware Large Kernel Convolutions 1 2 8 Feb 2020 official (archive's flag): 6 ran · 6 ran (of which 2 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 4 where Syntology's instrument failed) · 1 unverified (3 pointer-only for licence)
Non-autoregressive Translation with Disentangled Context Transformer 1 1 1 Jan 2020 not harvested
MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning 3 2 17 Nov 2019 official: not harvested · 0 ran · 1 unverified (1 pointer-only for licence)
Data Diversification: A Simple Strategy For Neural Machine Translation 2 1 5 Nov 2019 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified (4 pointer-only for licence)
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer 57 2 23 Oct 2019 21 ran (of which 0 constructed an object rather than computing a result; 20 with no instrument failure: 1 honoured, 0 violated, 19 with no contract checked; 1 where Syntology's instrument failed) · 10 unverified
FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow 2 10 5 Sep 2019 official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified
Adaptively Sparse Transformers 3 2 30 Aug 2019 official (archive's flag): 2 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified
Depth Growing for Neural Machine Translation 1 2 3 Jul 2019 not harvested
Improving Neural Language Modeling via Adversarial Training 1 1 10 Jun 2019 not harvested
KERMIT: Generative Insertion-Based Modeling for Sequences 0 1 4 Jun 2019 not harvested
Levenshtein Transformer 3 1 27 May 2019 not harvested
Joint Source-Target Self Attention with Locality Constraints 2 2 16 May 2019 not harvested
Deep Residual Output Layers for Neural Language Generation 1 1 14 May 2019 not harvested
Synchronous Bidirectional Neural Machine Translation 2 1 13 May 2019 not harvested
MASS: Masked Sequence to Sequence Pre-training for Language Generation 7 2 7 May 2019 not harvested
An Effective Approach to Unsupervised Machine Translation 1 4 4 Feb 2019 not harvested
Memory-Efficient Adaptive Optimization 4 1 30 Jan 2019 not harvested
The Evolved Transformer 3 6 30 Jan 2019 not harvested
Pay Less Attention with Lightweight and Dynamic Convolutions 3 4 29 Jan 2019 not harvested
Cross-lingual Language Model Pretraining 17 2 22 Jan 2019 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 1 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (1 pointer-only for licence)
Unsupervised Neural Machine Translation with SMT as Posterior Regularization 1 4 14 Jan 2019 official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 0 where Syntology's instrument failed) · 8 unverified
Neural Machine Translation with Adequacy-Oriented Learning 0 1 21 Nov 2018 not harvested
Modeling Localness for Self-Attention Networks 0 1 24 Oct 2018 not harvested
Fast and Simple Mixture of Softmaxes with BPE and Hybrid-LightRNN for Language Generation 1 2 25 Sep 2018 not harvested
FRAGE: Frequency-Agnostic Word Representation 2 1 18 Sep 2018 not harvested
Unsupervised Statistical Machine Translation 3 5 4 Sep 2018 not harvested
Understanding Back-Translation at Scale 3 2 28 Aug 2018 community repositories only · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified
Universal Transformers 8 1 10 Jul 2018 community repositories only · 17 ran (of which 8 constructed an object rather than computing a result; 17 with no instrument failure: 1 honoured, 0 violated, 16 with no contract checked; 0 where Syntology's instrument failed) · 8 unverified (24 pointer-only for licence)
Dense Information Flow for Neural Machine Translation 1 1 3 Jun 2018 not harvested
Scaling Neural Machine Translation 5 2 1 Jun 2018 not harvested
Accelerating Neural Transformer via an Average Attention Network 1 3 2 May 2018 official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 0 where Syntology's instrument failed) · 12 unverified
The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation 3 2 26 Apr 2018 not harvested
Phrase-Based & Neural Unsupervised Machine Translation 14 8 20 Apr 2018 not harvested
Self-Attention with Relative Position Representations 13 2 6 Mar 2018 community repositories only · 13 ran (of which 4 constructed an object rather than computing a result; 8 with no instrument failure: 2 honoured, 0 violated, 6 with no contract checked; 5 where Syntology's instrument failed) · 10 unverified (3 pointer-only for licence)
Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement 2 2 19 Feb 2018 official (archive's flag): 2 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (1 pointer-only for licence)
Deliberation Networks: Sequence Generation Beyond One-Pass Decoding 0 1 1 Dec 2017 not harvested
Non-Autoregressive Neural Machine Translation 2 2 7 Nov 2017 official (archive's flag): 1 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (1 pointer-only for licence)
Weighted Transformer Network for Machine Translation 5 2 6 Nov 2017 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 2 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (2 pointer-only for licence)
Unsupervised Neural Machine Translation 2 1 30 Oct 2017 official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 5 where Syntology's instrument failed) · 0 unverified (6 pointer-only for licence)
Simple Recurrent Units for Highly Parallelizable Recurrence 11 1 8 Sep 2017 community repositories only · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified
Attention Is All You Need 595 4 12 Jun 2017 official: no sample here; runs from other or unrecorded repositories · 610 ran (of which 293 constructed an object rather than computing a result; 529 with no instrument failure: 45 honoured, 15 violated, 469 with no contract checked; 81 where Syntology's instrument failed) · 336 unverified (451 pointer-only for licence)
Depthwise Separable Convolutions for Neural Machine Translation 2 1 9 Jun 2017 not harvested
Convolutional Sequence to Sequence Learning 37 4 8 May 2017 not harvested
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer 4 2 23 Jan 2017 4 ran (of which 4 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified; every one of the 4 samples that ran constructed an object rather than computing a result (6 pointer-only for licence)
A Convolutional Encoder Model for Neural Machine Translation 2 1 7 Nov 2016 not harvested
Neural Machine Translation in Linear Time 11 1 31 Oct 2016 not harvested
Can Active Memory Replace Attention? 2 1 27 Oct 2016 not harvested
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation 28 2 26 Sep 2016 34 ran (of which 4 constructed an object rather than computing a result; 25 with no instrument failure: 1 honoured, 2 violated, 22 with no contract checked; 9 where Syntology's instrument failed) · 12 unverified (13 pointer-only for licence)
Neural Semantic Encoders 3 1 14 Jul 2016 not harvested
Sequence-Level Knowledge Distillation 6 1 25 Jun 2016 official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified
Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation 1 3 14 Jun 2016 not harvested
Edinburgh's Syntax-Based Systems at WMT 2015 0 1 1 Sep 2015 not harvested
Effective Approaches to Attention-based Neural Machine Translation 44 3 17 Aug 2015 7 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 1 honoured, 0 violated, 5 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (2 pointer-only for licence)
Addressing the Rare Word Problem in Neural Machine Translation 5 1 30 Oct 2014 not harvested
Sequence to Sequence Learning with Neural Networks 74 2 10 Sep 2014 21 ran (of which 0 constructed an object rather than computing a result; 13 with no instrument failure: 3 honoured, 0 violated, 10 with no contract checked; 8 where Syntology's instrument failed) · 4 unverified (9 pointer-only for licence)
Recurrent Neural Network Regularization 21 1 8 Sep 2014 community repositories only · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 4 unverified (6 pointer-only for licence)
Neural Machine Translation by Jointly Learning to Align and Translate 124 1 1 Sep 2014 41 ran (of which 2 constructed an object rather than computing a result; 24 with no instrument failure: 4 honoured, 0 violated, 20 with no contract checked; 17 where Syntology's instrument failed) · 3 unverified (16 pointer-only for licence)
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation 42 1 3 Jun 2014 13 ran (of which 6 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 3 where Syntology's instrument failed) · 9 unverified (14 pointer-only for licence)

Dataset loaders archive 2025-07-28

4 loaders as listed in the archive; links are outbound and not re-checked here.

Tasks archive 2025-07-28

License archive 2025-07-28

Unknown

Modalities archive 2025-07-28

Languages archive 2025-07-28

Variants archive 2025-07-28

  • WMT-newstest2014-deen
  • WMT 2014 News
  • newstest2014-deen eng-deu
  • newstest2014-deen deu-eng
  • newstest2014-deen
  • wmt14
  • WMT 2014
  • WMT2014 German-English
  • WMT2014 English-Czech
  • WMT 2014 EN-FR
  • WMT 2014 EN-DE
  • WMT2014 English-German
  • WMT2014 French-English
  • WMT2014 English-French

14 variant names, as the archive lists them.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections