Methods › Natural Language Processing › Autoregressive Transformers › GPT-2
GPT-2
Introduced by Alec Radford et al. in Language Models are Unsupervised Multitask Learners
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
GPT-2 is a Transformer architecture that was notable for its size (1.5 billion parameters) on its release. The model is pretrained on a WebText dataset - text from 45 million website links. It largely follows the previous GPT architecture with some modifications:
-
Layer normalization is moved to the input of each sub-block, similar to a pre-activation residual network and an additional layer normalization was added after the final self-attention block.
-
A modified initialization which accounts for the accumulation on the residual path with model depth is used. Weights of residual layers are scaled at initialization by a factor of 1/√(N) where N is the number of residual layers.
-
The vocabulary is expanded to 50,257. The context size is expanded from 512 to 1024 tokens and a larger batch size of 512 is used.
Source in the archive: Language Models are Unsupervised Multitask Learners, a link on d4mucfpksywv.cloudfront.net (archive link, not checked and not linked: not a paper host this site links to).
Papers archive 2025-07-28
30 shown of 768, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements 27 Jun 2025 · 1 repository · arXiv:2506.22419
-
M2BeamLLM: Multimodal Sensing-empowered mmWave Beam Prediction with Large Language Models 17 Jun 2025 · 0 repositories · arXiv:2506.14532
-
Decomposing MLP Activations into Interpretable Features via Semi-Nonnegative Matrix Factorization 12 Jun 2025 · 1 repository · arXiv:2506.10920
-
A Novel Lightweight Transformer with Edge-Aware Fusion for Remote Sensing Image Captioning 11 Jun 2025 · 0 repositories · arXiv:2506.09429
-
Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Models 8 Jun 2025 · 0 repositories · arXiv:2506.07121
-
Dissecting Bias in LLMs: A Mechanistic Interpretability Perspective 5 Jun 2025 · 0 repositories · arXiv:2506.05166
-
An Exploratory Framework for Future SETI Applications: Detecting Generative Reactivity via Language Models 3 Jun 2025 · 0 repositories · arXiv:2506.02730
-
Rethinking the effects of data contamination in Code Intelligence 3 Jun 2025 · 0 repositories · arXiv:2506.02791
-
How Neural Networks Organize Concepts: Introducing Concept Trajectory Analysis for Deep Learning Interpretability 1 Jun 2025 · 1 repository
-
Power-of-Two (PoT) Weights in Large Language Models (LLMs) 31 May 2025 · 0 repositories · arXiv:2506.00315
-
Matryoshka Model Learning for Improved Elastic Student Models 29 May 2025 · 0 repositories · arXiv:2505.23337
-
Privacy-Preserving Chest X-ray Report Generation via Multimodal Federated Learning with ViT and GPT-2 27 May 2025 · 0 repositories · arXiv:2505.21715
-
Anveshana: A New Benchmark Dataset for Cross-Lingual Information Retrieval On English Queries and Sanskrit Documents 26 May 2025 · 0 repositories · arXiv:2505.19494
-
Conversational Lexicography: Querying Lexicographic Data on Knowledge Graphs with SPARQL through Natural Language 26 May 2025 · 0 repositories · arXiv:2505.19971
-
ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining 26 May 2025 · 0 repositories · arXiv:2505.19893
-
Strong Membership Inference Attacks on Massive Datasets and (Moderately) Large Language Models 24 May 2025 · 0 repositories · arXiv:2505.18773
-
AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training 22 May 2025 · 1 repository · arXiv:2505.16363Syntology ran 0 of 1 samples · 1 unverified
-
The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm 22 May 2025 · 0 repositories · arXiv:2505.16932
-
Breaking Bad Tokens: Detoxification of LLMs Using Sparse Autoencoders 20 May 2025 · 0 repositories · arXiv:2505.14536Syntology ran 2 of 2 samples · 0 unverified · 2 pointer-only (licence)
-
Out-of-Distribution Generalization of In-Context Learning: A Low-Dimensional Subspace Perspective 20 May 2025 · 0 repositories · arXiv:2505.14808
-
Scaling Laws for State Dynamics in Large Language Models 20 May 2025 · 0 repositories · arXiv:2505.14892
-
VesselGPT: Autoregressive Modeling of Vascular Geometry 19 May 2025 · 1 repository · arXiv:2505.13318
-
Can an Easy-to-Hard Curriculum Make Reasoning Emerge in Small Language Models? Evidence from a Four-Stage Curriculum on GPT-2 16 May 2025 · 0 repositories · arXiv:2505.11643
-
Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiency 16 May 2025 · 0 repositories · arXiv:2505.13499
-
Are Sparse Autoencoders Useful for Java Function Bug Detection? 15 May 2025 · 1 repository · arXiv:2505.10375
-
Memorization-Compression Cycles Improve Generalization 13 May 2025 · 0 repositories · arXiv:2505.08727
-
Probability Consistency in Large Language Models: Theoretical Foundations Meet Empirical Discrepancies 13 May 2025 · 1 repository · arXiv:2505.08739
-
LLM-e Guess: Can LLMs Capabilities Advance Without Hardware Progress? 7 May 2025 · 1 repository · arXiv:2505.04075
-
UniDetox: Universal Detoxification of Large Language Models via Dataset Distillation 29 Apr 2025 · 1 repository · arXiv:2504.20500Syntology ran 1 of 1 samples · 0 unverified
-
VEXP: A Low-Cost RISC-V ISA Extension for Accelerated Softmax Computation in Transformers 15 Apr 2025 · 0 repositories · arXiv:2504.11227
Tasks archive 2025-07-28
20 shown of 398 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
| Task | Papers |
|---|---|
| Language Modelling | 240 |
| Language Modeling | 183 |
| Text Generation | 128 |
| Sentence | 49 |
| Question Answering | 40 |
| Decoder | 39 |
| Large Language Model | 30 |
| Transfer Learning | 28 |
| GPU | 26 |
| Retrieval | 23 |
| Diversity | 20 |
| Word Embeddings | 20 |
| In-Context Learning | 19 |
| model | 19 |
| Data Augmentation | 18 |
| Text Classification | 17 |
| Translation | 17 |
| Response Generation | 16 |
| Sentiment Analysis | 16 |
| Reinforcement Learning (RL) | 15 |
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections