Datasets › GEM
GEM (Generation, Evaluation, and Metrics)
Generation, Evaluation, and Metrics (GEM) is a benchmark environment for Natural Language Generation with a focus on its Evaluation, both through human annotations and automated Metrics.
GEM aims to:
- measure NLG progress across 13 datasets spanning many NLG tasks and languages.
- provide an in-depth analysis of data and models presented via data statements and challenge sets.
- develop standards for evaluation of generated text using both automated and human metrics.
It is our goal to regularly update GEM and to encourage toward more inclusive practices in dataset development by extending existing data or developing datasets for additional languages.
Source: https://gem-benchmark.com/ Image Source: Gehrmann et al
Benchmarks archive 2025-07-28
All 1 leaderboard whose dataset resolves to this page shown (sort by any header). "First row" is the archive's own first row at snapshot, in the archive's row order; nothing here re-ranks and metric direction is not asserted.
| First row (archive order) | Paper | Code | ||||
|---|---|---|---|---|---|---|
| Extreme Summarization | GEM-XSum | PEGASUS ROUGE-2 23.2 | The GEM Benchmark: Natural Language Generation, its... | — | 6 | Compare |
Papers archive 2025-07-28
3 shown of 3 papers with a leaderboard row on this dataset's benchmarks, newest first. The archive's own "papers using this dataset" list was never published, so this is the benchmark-backed subset; the archive's count for this dataset is 45. The Syntology column is from Syntology's graph (read 2026-09-24), stated per sample; it is not part of any archive number.
| Date | Samples run Syntology | |||
|---|---|---|---|---|
| PaLM: Scaling Language Modeling with Pathways | 7 | 3 | 5 Apr 2022 | ran 30 of 37 samples (7 unverified) |
| ByT5: Towards a token-free future with pre-trained byte-to-byte models | 5 | 2 | 28 May 2021 | ran 0 of 6 samples (6 unverified) |
| The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics | 0 | 1 | 2 Feb 2021 | not harvested |
Dataset loaders archive 2025-07-28
3 loaders as listed in the archive; links are outbound and not re-checked here.
Tasks archive 2025-07-28
License archive 2025-07-28
No licence recorded in the archive. Absence here is not a statement about the dataset's terms.
Modalities archive 2025-07-28
Languages archive 2025-07-28
Variants archive 2025-07-28
- GEM
- GEM-XSum
2 variant names, as the archive lists them.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections