Papers › Improving Language Understanding by Generative Pre-Training
Improving Language Understanding by Generative Pre-Training
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever
Natural language understanding comprises a wide range of diverse tasks such as textual entailment, question answering, semantic similarity assessment, and document classification. Although large unlabeled text corpora are abundant, labeled data for learning these specific tasks is scarce, making it challenging for discriminatively trained models to perform adequately. We demonstrate that large gains on these tasks can be realized by generative pre-training of a language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task. In contrast to previous approaches, we make use of task-aware input transformations during fine-tuning to achieve effective transfer while requiring minimal changes to the model architecture. We demonstrate the effectiveness of our approach on a wide range of benchmarks for natural language understanding. Our general task-agnostic model outperforms discriminatively trained models that use architectures specifically crafted for each task, significantly improving upon the state of the art in 9 out of the 12 tasks studied. For instance, we achieve absolute improvements of 8.9% on commonsense reasoning (Stories Cloze Test), 5.7% on question answering (RACE), and 1.5% on textual entailment (MultiNLI).
Code
13 repositories listed; official and paper-mentioned ones first.
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Natural Language Inference | MultiNLI | Finetuned Transformer LM | Matched | 82.1 | #41 of 67 | Archive leaderboard | report |
| Natural Language Inference | MultiNLI | Finetuned Transformer LM | Mismatched | 81.4 | #41 of 67 | Archive leaderboard | report |
| Natural Language Inference | SNLI | Fine-Tuned LM-Pretrained Transformer | % Test Accuracy | 89.9 | #15 of 98 | Archive leaderboard | report |
| Natural Language Inference | SNLI | Fine-Tuned LM-Pretrained Transformer | % Train Accuracy | 96.6 | #15 of 98 | Archive leaderboard | report |
| Natural Language Inference | SNLI | Fine-Tuned LM-Pretrained Transformer | Parameters | 85m | #15 of 98 | Archive leaderboard | report |
| Natural Language Inference | SciTail | Finetuned Transformer LM | Accuracy | 88.3 | #4 of 13 | Archive leaderboard | report |
| Question Answering | RACE | Finetuned Transformer LM | RACE | 59.0 | #4 of 7 | Archive leaderboard | report |
| Question Answering | RACE | Finetuned Transformer LM | RACE-h | 57.4 | #4 of 7 | Archive leaderboard | report |
| Question Answering | RACE | Finetuned Transformer LM | RACE-m | 62.9 | #4 of 7 | Archive leaderboard | report |
| Question Answering | StoryCloze | Finetuned Transformer LM | Accuracy | 86.5 | #8 of 23 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Introduced by this paper: GPT
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections