Papers › Neural Grammatical Error Correction Systems with Unsupervised Pre-training on Synthetic Data

Neural Grammatical Error Correction Systems with Unsupervised Pre-training on Synthetic Data

1 Aug 2019WS 2019 8archive 2025-07-28

Roman Grundkiewicz, Marcin Junczys-Dowmunt, Kenneth Heafield

Considerable effort has been made to address the data sparsity problem in neural grammatical error correction. In this work, we propose a simple and surprisingly effective unsupervised synthetic error generation method based on confusion sets extracted from a spellchecker to increase the amount of training data. Synthetic data is used to pre-train a Transformer sequence-to-sequence model, which not only improves over a strong baseline trained on authentic error-annotated data, but also enables the development of a practical GEC system in a scenario where little genuine error-annotated data is available. The developed systems placed first in the BEA19 shared task, achieving 69.47 and 64.24 F_(0.5) in the restricted and low-resource tracks respectively, both on the W{\&}I+LOCNESS test set. On the popular CoNLL 2014 test set, we report state-of-the-art results of 64.16 M{\mbox{²}} for the submitted system, and 61.30 M{\mbox{²}} for the constrained system trained on the NUCLE and Lang-8 data.

Paper PDF Code

Code

grammatical/pretraining-bea2019 officialmentioned in paper report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Grammatical Error Correction Unsupervised Pre-training

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Grammatical Error Correction	BEA-2019 (test)	Transformer	F0.5	69.5	#16 of 19	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Absolute Position Encodings Adam Attention BPE Dense Connections Dropout Label Smoothing Layer Normalization Linear Layer Multi-Head Attention Position-Wise Feed-Forward Layer ReLU Residual Connection Softmax Transformer

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections