Papers › Ensembling and Knowledge Distilling of Large Sequence Taggers for Grammatical Error Correction

Ensembling and Knowledge Distilling of Large Sequence Taggers for Grammatical Error Correction

24 Mar 2022ACL 2022 5arXiv:2203.13064archive 2025-07-28

Maksym Tarnavskyi, Artem Chernodub, Kostiantyn Omelianchuk

In this paper, we investigate improvements to the GEC sequence tagging architecture with a focus on ensembling of recent cutting-edge Transformer-based encoders in Large configurations. We encourage ensembling models by majority votes on span-level edits because this approach is tolerant to the model architecture and vocabulary size. Our best ensemble achieves a new SOTA result with an F_(0.5) score of 76.05 on BEA-2019 (test), even without pre-training on synthetic datasets. In addition, we perform knowledge distillation with a trained ensemble to generate new synthetic training datasets, "Troy-Blogs" and "Troy-1BW". Our best single sequence tagging model that is pretrained on the generated Troy-datasets in combination with the publicly available synthetic PIE dataset achieves a near-SOTA (To the best of our knowledge, our best single model gives way only to much heavier T5 model result with an F_(0.5) score of 73.21 on BEA-2019 (test). The code, datasets, and trained models are publicly available).

Paper PDF Conference PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

makstarnavskyi/gector-large officialmentioned in papermentioned on GitHubpytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Grammatical Error Correction Knowledge Distillation

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Grammatical Error Correction	BEA-2019 (test)	DeBERTa + RoBERTa + XLNet	F0.5	76.05	#7 of 19	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Adafactor Attention Attention Dropout BPE Dense Connections Dropout Gated Linear Unit Inverse Square Root Schedule Knowledge Distillation Layer Normalization Linear Layer Multi-Head Attention Residual Connection SentencePiece Softmax T5

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections