Datasets › WikiText-103

WikiText-103

Introduced by Stephen Merity et al. in Pointer Sentinel Mixture Models26 Sep 2016 archive 2025-07-28

The WikiText language modeling dataset is a collection of over 100 million tokens extracted from the set of verified Good and Featured articles on Wikipedia. The dataset is available under the Creative Commons Attribution-ShareAlike License.

Compared to the preprocessed version of Penn Treebank (PTB), WikiText-2 is over 2 times larger and WikiText-103 is over 110 times larger. The WikiText dataset also features a far larger vocabulary and retains the original case, punctuation and numbers - all of which are removed in PTB. As it is composed of full articles, the dataset is well suited for models that can take advantage of long term dependencies.

Source: The WikiText Long Term Dependency Language Modeling Dataset Image Source: https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/

Benchmarks archive 2025-07-28

All 2 leaderboards whose dataset resolves to this page shown (sort by any header). "First row" is the archive's own first row at snapshot, in the archive's row order; nothing here re-ranks and metric direction is not asserted.

Syntology 6 newer papers were judged to report on this dataset's benchmarks so far (Claude Sonnet 4.5, reading their tables: a model's judgement, not a result; 0 placed): Language Modelling · WikiText-103 (6 judged, 0 placed). Drawn from 8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read).

First row (archive order)PaperCode
Language Modelling WikiText-103 RETRO (7.5B) Test perplexity 2.4 Improving language models by retrieving from trillions of tokens labmlai/annotated_deep_learning_paper_implementations +1 89 Compare
Text Generation WikiText-103 no rows — — 0 Compare

Papers archive 2025-07-28

papers with a benchmark row: 55 · with a code link: 51 · where Syntology ran a sample: 31 (28 with a run with no instrument failure, 3 where every run was a failure of Syntology's instrument) Syntology

Show: all papers with a benchmark rowonly where code ran (31 of 55 with a benchmark row: 28 with a run with no instrument failure, 3 where every run was a failure of Syntology's instrument)

Page 1 of 1: papers 1 to 55 of 55 with a leaderboard row on this dataset's benchmarks, newest first by the archive's date (ties by slug; undated papers last).

The archive's own "papers using this dataset" list was never published, so this is the benchmark-backed subset, not that list; the archive's count for this dataset is 560. The Syntology column is from Syntology's graph, stated per sample; it is not part of any archive number. A line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified” (C is a part of N, never taken away from it); the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. When the archive marks a repository official for the paper, the cell starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record.

PaperCodeResultsDateSamples run Syntology
Advancing State of the Art in Language Modeling 1 1 28 Nov 2023 not harvested
Memory-efficient Stochastic methods for Memory-based Transformers 1 1 14 Nov 2023 not harvested
GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling 3 1 3 Nov 2023 official (archive's flag): 2 ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 1 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified
The Information Pathways Hypothesis: Transformers are Dynamic Self-Ensembles 1 2 2 Jun 2023 not harvested
Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representation 1 1 31 May 2023 official (archive's flag): 1 ran · 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result (1 pointer-only for licence)
Hyena Hierarchy: Towards Larger Convolutional Language Models 7 2 21 Feb 2023 official (archive's flag): 2 ran · 5 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 5 where Syntology's instrument failed) · 0 unverified (4 pointer-only for licence)
Hungry Hungry Hippos: Towards Language Modeling with State Space Models 3 5 28 Dec 2022 official (archive's flag): 1 ran · 7 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 4 where Syntology's instrument failed) · 8 unverified
You can't pick your neighbors, or can you? When and how to rely on retrieval in the kNN-LM 1 1 28 Oct 2022 official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (3 pointer-only for licence)
Mega: Moving Average Equipped Gated Attention 7 1 21 Sep 2022 official (archive's flag): 3 ran · 12 ran (of which 6 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 5 where Syntology's instrument failed) · 1 unverified (11 pointer-only for licence)
General-purpose, long-context autoregressive modeling with Perceiver AR 3 1 15 Feb 2022 official (archive's flag): 1 ran · 9 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 4 where Syntology's instrument failed) · 3 unverified (1 pointer-only for licence)
Improving language models by retrieving from trillions of tokens 2 1 8 Dec 2021 16 ran (of which 5 constructed an object rather than computing a result; 14 with no instrument failure: 0 honoured, 3 violated, 11 with no contract checked; 2 where Syntology's instrument failed) · 7 unverified (3 pointer-only for licence)
Efficiently Modeling Long Sequences with Structured State Spaces 8 1 31 Oct 2021 official (archive's flag): 5 ran · 37 ran (of which 8 constructed an object rather than computing a result; 21 with no instrument failure: 2 honoured, 1 violated, 18 with no contract checked; 16 where Syntology's instrument failed) · 18 unverified (3 pointer-only for licence)
∞-former: Infinite Memory Transformer 1 7 1 Sep 2021 not harvested
FNetAR: Mixing Tokens with Autoregressive Fourier Transforms 1 1 22 Jul 2021 not harvested
Differentiable Model Compression via Pseudo Quantization Noise 1 1 20 Apr 2021 not harvested
Revisiting Simple Neural Probabilistic Language Models 1 1 8 Apr 2021 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified
Finetuning Pretrained Transformers into RNNs 2 1 24 Mar 2021 3 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 5 unverified (4 pointer-only for licence)
GLM: General Language Model Pretraining with Autoregressive Blank Infilling 8 2 18 Mar 2021 community repositories only · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified
Random Feature Attention 0 2 3 Mar 2021 not harvested
When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute 1 2 24 Feb 2021 not harvested
Subformer: A Parameter Reduced Transformer 0 1 1 Jan 2021 not harvested
Shortformer: Better Language Modeling using Shorter Inputs 1 2 31 Dec 2020 not harvested
Rethinking Attention with Performers 7 1 30 Sep 2020 official: harvested, nothing ran · 11 ran (of which 3 constructed an object rather than computing a result; 7 with no instrument failure: 1 honoured, 0 violated, 6 with no contract checked; 4 where Syntology's instrument failed) · 5 unverified (6 pointer-only for licence)
Pay Attention when Required 2 2 9 Sep 2020 not harvested
DeLighT: Deep and Light-weight Transformer 2 1 3 Aug 2020 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention 8 1 29 Jun 2020 official: no sample here; runs from other or unrecorded repositories · 6 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified (3 pointer-only for licence)
How much complexity does an RNN architecture need to learn syntax-sensitive dependencies? 1 3 17 May 2020 not harvested
Segatron: Segment-Aware Transformer for Language Modeling and Understanding 1 1 30 Apr 2020 not harvested
Efficient Content-Based Sparse Attention with Routing Transformers 2 1 12 Mar 2020 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 2 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (2 pointer-only for licence)
Addressing Some Limitations of Transformers with Feedback Memory 4 2 21 Feb 2020 official: no sample here; runs from other or unrecorded repositories · 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 2 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (2 pointer-only for licence)
Time-aware Large Kernel Convolutions 1 1 8 Feb 2020 official (archive's flag): 6 ran · 6 ran (of which 2 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 4 where Syntology's instrument failed) · 1 unverified (3 pointer-only for licence)
Reformer: The Efficient Transformer 10 1 13 Jan 2020 community repositories only · 6 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 5 where Syntology's instrument failed) · 2 unverified
Compressive Transformers for Long-Range Sequence Modelling 6 1 13 Nov 2019 6 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 1 where Syntology's instrument failed) · 5 unverified
Improving Transformer Models by Reordering their Sublayers 2 1 10 Nov 2019 not harvested
Generalization through Memorization: Nearest Neighbor Language Models 5 2 1 Nov 2019 community repositories only · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified
On the adequacy of untuned warmup for adaptive optimization 1 1 9 Oct 2019 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified
Alleviating Sequence Information Loss with Data Overlapping and Prime Batch Sizes 1 1 18 Sep 2019 not harvested
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism 10 1 17 Sep 2019 community repositories only · 12 ran (of which 2 constructed an object rather than computing a result; 7 with no instrument failure: 4 honoured, 0 violated, 3 with no contract checked; 5 where Syntology's instrument failed) · 35 unverified (15 pointer-only for licence)
Deep Equilibrium Models 11 3 3 Sep 2019 community repositories only · 9 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 1 honoured, 0 violated, 6 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified (4 pointer-only for licence)
Augmenting Self-attention with Persistent Memory 2 1 2 Jul 2019 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 3 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (4 pointer-only for licence)
Improving Neural Language Modeling via Adversarial Training 1 1 10 Jun 2019 not harvested
Improving Neural Language Models by Segmenting, Attending, and Predicting the Future 1 1 4 Jun 2019 not harvested
Language Models with Transformers 1 1 20 Apr 2019 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (2 pointer-only for licence)
Dynamic Evaluation of Transformer Language Models 1 2 17 Apr 2019 not harvested
Language Models are Unsupervised Multitask Learners 21 4 14 Feb 2019 not harvested
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context 37 2 9 Jan 2019 official (archive's flag): 8 ran · 67 ran (of which 37 constructed an object rather than computing a result; 49 with no instrument failure: 4 honoured, 1 violated, 44 with no contract checked; 18 where Syntology's instrument failed) · 76 unverified (43 pointer-only for licence)
Trellis Networks for Sequence Modeling 1 1 15 Oct 2018 official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 2 honoured, 0 violated, 4 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (2 pointer-only for licence)
Adaptive Input Representations for Neural Language Modeling 3 1 28 Sep 2018 not harvested
Relational recurrent neural networks 2 1 5 Jun 2018 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (1 pointer-only for licence)
Fast Parametric Learning with Activation Memorization 0 4 27 Mar 2018 not harvested
An Analysis of Neural Language Modeling at Multiple Scales 12 1 22 Mar 2018 community repositories only · 14 ran (of which 0 constructed an object rather than computing a result; 13 with no instrument failure: 2 honoured, 0 violated, 11 with no contract checked; 1 where Syntology's instrument failed) · 4 unverified (4 pointer-only for licence)
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling 35 1 4 Mar 2018 community repositories only · 9 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 0 honoured, 1 violated, 7 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (1 pointer-only for licence)
Convolutional Sequence Modeling Revisited 0 1 1 Jan 2018 not harvested
Language Modeling with Gated Convolutional Networks 11 2 23 Dec 2016 not harvested
Improving Neural Language Models with a Continuous Cache 14 3 13 Dec 2016 not harvested

Dataset loaders archive 2025-07-28

1 loader as listed in the archive; links are outbound and not re-checked here.

Tasks archive 2025-07-28

License archive 2025-07-28

CC BY-SA 3.0

Modalities archive 2025-07-28

Languages archive 2025-07-28

Variants archive 2025-07-28

  • WikiText-103

1 variant name, as the archive lists them.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections