Datasets › OpenWebText
OpenWebText
OpenWebText is an open-source recreation of the WebText corpus. The text is web content extracted from URLs shared on Reddit with at least three upvotes. (38GB).
Source: RoBERTa: A Robustly Optimized BERT Pretraining Approach
Benchmarks archive 2025-07-28
All 2 leaderboards whose dataset resolves to this page shown (sort by any header). "First row" is the archive's own first row at snapshot, in the archive's row order; nothing here re-ranks and metric direction is not asserted.
| First row (archive order) | Paper | Code | ||||
|---|---|---|---|---|---|---|
| Language Modelling | OpenWebText | MDLM-Prime eval_perplexity 15.36 | Beyond Masked and Unmasked: Discrete Diffusion Models... | — | 12 | Compare |
| Text Generation | OpenWebText | GPT2-Hermite eval_loss 2.91 | Polynomial, trigonometric, and tropical activations | K-H-Ismail/torchortho | 3 | Compare |
Papers archive 2025-07-28
9 shown of 9 papers with a leaderboard row on this dataset's benchmarks, newest first. The archive's own "papers using this dataset" list was never published, so this is the benchmark-backed subset; the archive's count for this dataset is 207. The Syntology column is from Syntology's graph (read 2026-09-24), stated per sample; it is not part of any archive number.
| Date | Samples run Syntology | |||
|---|---|---|---|---|
| Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking | 0 | 1 | 24 May 2025 | not harvested |
| Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models | 2 | 1 | 12 Mar 2025 | ran 7 of 15 samples (8 unverified) |
| Polynomial, trigonometric, and tropical activations | 1 | 5 | 3 Feb 2025 | not harvested |
| Energy-Based Diffusion Language Models for Text Generation | 0 | 2 | 28 Oct 2024 | not harvested |
| Loop Neural Networks for Parameter Sharing | 0 | 1 | 21 Sep 2024 | not harvested |
| Simple and Effective Masked Diffusion Language Models | 2 | 2 | 11 Jun 2024 | ran 1 of 1 samples (0 unverified; 1 pointer-only for licence) |
| Simplified and Generalized Masked Diffusion for Discrete Data | 1 | 1 | 6 Jun 2024 | ran 6 of 21 samples (15 unverified) |
| Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution | 4 | 1 | 25 Oct 2023 | ran 13 of 18 samples (5 unverified) |
| Language Models are Unsupervised Multitask Learners | 21 | 1 | 14 Feb 2019 | not harvested |
Dataset loaders archive 2025-07-28
3 loaders as listed in the archive; links are outbound and not re-checked here.
Tasks archive 2025-07-28
License archive 2025-07-28
Modalities archive 2025-07-28
Languages archive 2025-07-28
Variants archive 2025-07-28
- OpenWebText
1 variant name, as the archive lists them.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections