Papers › Improving Language Understanding by Generative Pre-Training

Improving Language Understanding by Generative Pre-Training

11 Jun 2018Preprint 2018 6archive 2025-07-28

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever

Natural language understanding comprises a wide range of diverse tasks such as textual entailment, question answering, semantic similarity assessment, and document classification. Although large unlabeled text corpora are abundant, labeled data for learning these specific tasks is scarce, making it challenging for discriminatively trained models to perform adequately. We demonstrate that large gains on these tasks can be realized by generative pre-training of a language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task. In contrast to previous approaches, we make use of task-aware input transformations during fine-tuning to achieve effective transfer while requiring minimal changes to the model architecture. We demonstrate the effectiveness of our approach on a wide range of benchmarks for natural language understanding. Our general task-agnostic model outperforms discriminatively trained models that use architectures specifically crafted for each task, significantly improving upon the state of the art in 9 out of the 12 tasks studied. For instance, we achieve absolute improvements of 8.9% on commonsense reasoning (Stories Cloze Test), 5.7% on question answering (RACE), and 1.5% on textual entailment (MultiNLI).

PaperPDFCode

Syntology This paper has no arXiv id in the archive, and this site joins papers to Syntology's graph by arXiv id, so this page does not say whether the graph holds it.

Code

13 repositories listed; official and paper-mentioned ones first.

2023-MindSpore-1/ms-code-152 mindsporeSyntology: not harvested report
Mind23-2/MindCode-139 mindsporeSyntology: not harvested report
MindSpore-scientific/code-7 Syntology: not harvested report
PaddlePaddle/FleetX paddleSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System... (arXiv:2205.10034; that paper's own run record: 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
abhaskumarsinha/Corpus2GPT pytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Language Models are Few-Shot Learners (arXiv:2005.14165; that paper's own run record: 45 ran (of which 0 constructed an object rather than computing a result; 37 with no instrument failure: 3 honoured, 2 violated, 32 with no contract checked; 8 where Syntology's instrument failed) · 20 unverified) and LLaMA: Open and Efficient Foundation Language Models (arXiv:2302.13971; that paper's own run record: 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified); nothing checks that they implement this paper's method. report
abhaskumarsinha/Keras-implementation-of-Transformer-Architecture tfSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Attention Is All You Need (arXiv:1706.03762; that paper's own run record: 610 ran (of which 293 constructed an object rather than computing a result; 529 with no instrument failure: 45 honoured, 15 violated, 469 with no contract checked; 81 where Syntology's instrument failed) · 336 unverified); nothing checks that they implement this paper's method. report
abhaskumarsinha/MinimalGPT tfSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Attention Is All You Need (arXiv:1706.03762; that paper's own run record: 610 ran (of which 293 constructed an object rather than computing a result; 529 with no instrument failure: 45 honoured, 15 violated, 469 with no contract checked; 81 where Syntology's instrument failed) · 336 unverified) and Language Models are Few-Shot Learners (arXiv:2005.14165; that paper's own run record: 45 ran (of which 0 constructed an object rather than computing a result; 37 with no instrument failure: 3 honoured, 2 violated, 32 with no contract checked; 8 where Syntology's instrument failed) · 20 unverified); nothing checks that they implement this paper's method. report
huggingface/transformers pytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Attention Is All You Need (arXiv:1706.03762; that paper's own run record: 610 ran (of which 293 constructed an object rather than computing a result; 529 with no instrument failure: 45 honoured, 15 violated, 469 with no contract checked; 81 where Syntology's instrument failed) · 336 unverified), HuggingFace's Transformers: State-of-the-art Natural Language Processing (arXiv:1910.03771; that paper's own run record: 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified), DeBERTa: Decoding-enhanced BERT with Disentangled Attention (arXiv:2006.03654; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 9 unverified) and 1 more (every one in this page's JSON twin); nothing checks that they implement this paper's method. report
lattice-ai/GPT tfSyntology: not harvested report
lvyufeng/bert4ms mindsporeApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to RoBERTa: A Robustly Optimized BERT Pretraining Approach (arXiv:1907.11692; that paper's own run record: 37 ran (of which 11 constructed an object rather than computing a result; 36 with no instrument failure: 2 honoured, 0 violated, 34 with no contract checked; 1 where Syntology's instrument failed) · 11 unverified); nothing checks that they implement this paper's method. report
milmor/GPT tfMITSyntology: not harvested report
openai/finetune-transformer-lm tfSyntology: not harvested report
xiuyu0000/new_papers_codes mindsporeSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Cloze TestDocument ClassificationLanguage ModelingLanguage ModellingNatural Language InferenceNatural Language UnderstandingQuestion AnsweringSemantic SimilaritySemantic Textual Similarity

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Natural Language Inference MultiNLI Finetuned Transformer LM Matched 82.1 #41 of 67 Archive leaderboard report
Natural Language Inference MultiNLI Finetuned Transformer LM Mismatched 81.4 #41 of 67 Archive leaderboard report
Natural Language Inference SNLI Fine-Tuned LM-Pretrained Transformer % Test Accuracy 89.9 #15 of 98 Archive leaderboard report
Natural Language Inference SNLI Fine-Tuned LM-Pretrained Transformer % Train Accuracy 96.6 #15 of 98 Archive leaderboard report
Natural Language Inference SNLI Fine-Tuned LM-Pretrained Transformer Parameters 85m #15 of 98 Archive leaderboard report
Natural Language Inference SciTail Finetuned Transformer LM Accuracy 88.3 #4 of 13 Archive leaderboard report
Question Answering RACE Finetuned Transformer LM RACE 59.0 #4 of 7 Archive leaderboard report
Question Answering RACE Finetuned Transformer LM RACE-h 57.4 #4 of 7 Archive leaderboard report
Question Answering RACE Finetuned Transformer LM RACE-m 62.9 #4 of 7 Archive leaderboard report
Question Answering StoryCloze Finetuned Transformer LM Accuracy 86.5 #8 of 23 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Introduced by this paper: GPT

AdamAttentionAttention DropoutBPECosine AnnealingDense ConnectionsDiscriminative Fine-TuningDropoutGPTLayer NormalizationLinear LayerLinear Warmup With Cosine AnnealingMulti-Head AttentionResidual ConnectionSoftmaxWeight Decay

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections