Methods › Natural Language Processing › Autoregressive Transformers › GPT-2
GPT-2
Introduced by Alec Radford et al. in Language Models are Unsupervised Multitask Learners
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
GPT-2 is a Transformer architecture that was notable for its size (1.5 billion parameters) on its release. The model is pretrained on a WebText dataset - text from 45 million website links. It largely follows the previous GPT architecture with some modifications:
-
Layer normalization is moved to the input of each sub-block, similar to a pre-activation residual network and an additional layer normalization was added after the final self-attention block.
-
A modified initialization which accounts for the accumulation on the residual path with model depth is used. Weights of residual layers are scaled at initialization by a factor of 1/√(N) where N is the number of residual layers.
-
The vocabulary is expanded to 50,257. The context size is expanded from 512 to 1024 tokens and a larger batch size of 512 is used.
Source in the archive: Language Models are Unsupervised Multitask Learners, a link on d4mucfpksywv.cloudfront.net (archive link, not checked and not linked: not a paper host this site links to).
In Syntology name not resolved (unknown_name). The lookup matched no method, paper or venue by this name; that asserts nothing about whether code exists.
Near matches the lookup lists, each a different name: GPT; GPT-D; H-GPT; RF-GPT; MS-GPT; GPT-RE; VA-GPT; GPT-ST.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 768 · with a code link: 339 · where Syntology ran a sample: 125 (98 with a run with no instrument failure, 27 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (125 of 768 tagged: 98 with a run with no instrument failure, 27 where every run was a failure of Syntology's instrument)
Page 1 of 8: papers 1 to 100 of 768, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements 27 Jun 2025 · 1 repository · arXiv:2506.22419
-
M2BeamLLM: Multimodal Sensing-empowered mmWave Beam Prediction with Large Language Models 17 Jun 2025 · 0 repositories · arXiv:2506.14532
-
Decomposing MLP Activations into Interpretable Features via Semi-Nonnegative Matrix Factorization 12 Jun 2025 · 1 repository · arXiv:2506.10920
-
A Novel Lightweight Transformer with Edge-Aware Fusion for Remote Sensing Image Captioning 11 Jun 2025 · 0 repositories · arXiv:2506.09429
-
Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Models 8 Jun 2025 · 0 repositories · arXiv:2506.07121
-
Dissecting Bias in LLMs: A Mechanistic Interpretability Perspective 5 Jun 2025 · 0 repositories · arXiv:2506.05166
-
An Exploratory Framework for Future SETI Applications: Detecting Generative Reactivity via Language Models 3 Jun 2025 · 0 repositories · arXiv:2506.02730
-
Rethinking the effects of data contamination in Code Intelligence 3 Jun 2025 · 0 repositories · arXiv:2506.02791
-
How Neural Networks Organize Concepts: Introducing Concept Trajectory Analysis for Deep Learning Interpretability 1 Jun 2025 · 1 repository
-
Power-of-Two (PoT) Weights in Large Language Models (LLMs) 31 May 2025 · 0 repositories · arXiv:2506.00315
-
Matryoshka Model Learning for Improved Elastic Student Models 29 May 2025 · 0 repositories · arXiv:2505.23337
-
Privacy-Preserving Chest X-ray Report Generation via Multimodal Federated Learning with ViT and GPT-2 27 May 2025 · 0 repositories · arXiv:2505.21715
-
Anveshana: A New Benchmark Dataset for Cross-Lingual Information Retrieval On English Queries and Sanskrit Documents 26 May 2025 · 0 repositories · arXiv:2505.19494
-
Conversational Lexicography: Querying Lexicographic Data on Knowledge Graphs with SPARQL through Natural Language 26 May 2025 · 0 repositories · arXiv:2505.19971
-
ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining 26 May 2025 · 0 repositories · arXiv:2505.19893
-
Strong Membership Inference Attacks on Massive Datasets and (Moderately) Large Language Models 24 May 2025 · 0 repositories · arXiv:2505.18773
-
AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training 22 May 2025 · 1 repository · arXiv:2505.16363Syntology official: no sample here; runs from other or unrecorded repositories · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm 22 May 2025 · 0 repositories · arXiv:2505.16932
-
Breaking Bad Tokens: Detoxification of LLMs Using Sparse Autoencoders 20 May 2025 · 0 repositories · arXiv:2505.14536Syntology 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
Out-of-Distribution Generalization of In-Context Learning: A Low-Dimensional Subspace Perspective 20 May 2025 · 0 repositories · arXiv:2505.14808
-
Scaling Laws for State Dynamics in Large Language Models 20 May 2025 · 0 repositories · arXiv:2505.14892
-
VesselGPT: Autoregressive Modeling of Vascular Geometry 19 May 2025 · 1 repository · arXiv:2505.13318
-
Can an Easy-to-Hard Curriculum Make Reasoning Emerge in Small Language Models? Evidence from a Four-Stage Curriculum on GPT-2 16 May 2025 · 0 repositories · arXiv:2505.11643
-
Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiency 16 May 2025 · 0 repositories · arXiv:2505.13499
-
Are Sparse Autoencoders Useful for Java Function Bug Detection? 15 May 2025 · 1 repository · arXiv:2505.10375
-
Memorization-Compression Cycles Improve Generalization 13 May 2025 · 0 repositories · arXiv:2505.08727
-
Probability Consistency in Large Language Models: Theoretical Foundations Meet Empirical Discrepancies 13 May 2025 · 1 repository · arXiv:2505.08739
-
LLM-e Guess: Can LLMs Capabilities Advance Without Hardware Progress? 7 May 2025 · 1 repository · arXiv:2505.04075
-
UniDetox: Universal Detoxification of Large Language Models via Dataset Distillation 29 Apr 2025 · 1 repository · arXiv:2504.20500Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
VEXP: A Low-Cost RISC-V ISA Extension for Accelerated Softmax Computation in Transformers 15 Apr 2025 · 0 repositories · arXiv:2504.11227
-
ConceptFormer: Towards Efficient Use of Knowledge-Graph Embeddings in Large Language Models 10 Apr 2025 · 0 repositories · arXiv:2504.07624
-
Multimodal Cinematic Video Synthesis Using Text-to-Image and Audio Generation Models 6 Apr 2025 · 0 repositories · arXiv:2506.10005
-
Localized Definitions and Distributed Reasoning: A Proof-of-Concept Mechanistic Interpretability Study via Activation Patching 3 Apr 2025 · 1 repository · arXiv:2504.02976
-
Analysis of an Idealized Stochastic Polyak Method and its Application to Black-Box Model Distillation 2 Apr 2025 · 0 repositories · arXiv:2504.01898
-
Towards Interpretable Soft Prompts 2 Apr 2025 · 1 repository · arXiv:2504.02144
-
Accelerating High-Efficiency Organic Photovoltaic Discovery via Pretrained Graph Neural Networks and Generative Reinforcement Learning 31 Mar 2025 · 0 repositories · arXiv:2503.23766
-
The realization of tones in spontaneous spoken Taiwan Mandarin: a corpus-based survey and theory-driven computational modeling 29 Mar 2025 · 0 repositories · arXiv:2503.23163
-
GPT Meets Graphs and KAN Splines: Testing Novel Frameworks on Multitask Fine-Tuned GPT-2 with LoRA 25 Mar 2025 · 0 repositories · arXiv:2504.10490
-
GraPLUS: Graph-based Placement Using Semantics for Image Composition 20 Mar 2025 · 0 repositories · arXiv:2503.15761
-
VLog: Video-Language Models by Generative Retrieval of Narration Vocabulary 12 Mar 2025 · 1 repository · arXiv:2503.09402Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified (of 10 harvested samples) · 10 pointer-only (licence)
-
Interpretable and Robust Dialogue State Tracking via Natural Language Summarization with LLMs 11 Mar 2025 · 0 repositories · arXiv:2503.08857
-
Implicit Reasoning in Transformers is Reasoning through Shortcuts 10 Mar 2025 · 1 repository · arXiv:2503.07604Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 7 harvested samples) · 1 pointer-only (licence)
-
Improving cognitive diagnostics in pathology: a deep learning approach for augmenting perceptional understanding of histopathology images 10 Mar 2025 · 0 repositories · arXiv:2503.06894
-
The Box is in the Pen: Evaluating Commonsense Reasoning in Neural Machine Translation 5 Mar 2025 · 1 repository · arXiv:2503.03308
-
Effectively Steer LLM To Follow Preference via Building Confident Directions 4 Mar 2025 · 0 repositories · arXiv:2503.02989
-
Weak-to-Strong Generalization Even in Random Feature Networks, Provably 4 Mar 2025 · 0 repositories · arXiv:2503.02877
-
Attention Condensation via Sparsity Induced Regularized Training 3 Mar 2025 · 0 repositories · arXiv:2503.01564
-
CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation 28 Feb 2025 · 1 repository · arXiv:2502.21074Syntology 0 ran · 2 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
SeisMoLLM: Advancing Seismic Monitoring via Cross-modal Transfer with Pre-trained Large Language Model 27 Feb 2025 · 1 repository · arXiv:2502.19960Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified (of 6 harvested samples)
-
The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training 26 Feb 2025 · 0 repositories · arXiv:2502.19002
-
A Close Look at Decomposition-based XAI-Methods for Transformer Language Models 21 Feb 2025 · 2 repositories · arXiv:2502.15886
-
BP-GPT: Auditory Neural Decoding Using fMRI-prompted LLM 21 Feb 2025 · 1 repository · arXiv:2502.15172Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)
-
Entropy-UID: A Method for Optimizing Information Density 20 Feb 2025 · 0 repositories · arXiv:2502.14366
-
PitVQA++: Vector Matrix-Low-Rank Adaptation for Open-Ended Visual Question Answering in Pituitary Surgery 19 Feb 2025 · 1 repository · arXiv:2502.14149
-
AdaSplash: Adaptive Sparse Flash Attention 17 Feb 2025 · 1 repository · arXiv:2502.12082Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples)
-
AI-generated Text Detection with a GLTR-based Approach 17 Feb 2025 · 0 repositories · arXiv:2502.12064
-
Mechanistic Unveiling of Transformer Circuits: Self-Influence as a Key to Model Reasoning 13 Feb 2025 · 0 repositories · arXiv:2502.09022
-
InTAR: Inter-Task Auto-Reconfigurable Accelerator Design for High Data Volume Variation in DNNs 12 Feb 2025 · 1 repository · arXiv:2502.08807
-
YNote: A Novel Music Notation for Fine-Tuning LLMs in Music Generation 12 Feb 2025 · 0 repositories · arXiv:2502.10467
-
Find Central Dogma Again: Leveraging Multilingual Transfer in Large Language Models 10 Feb 2025 · 0 repositories · arXiv:2502.06253
-
Emergence of Episodic Memory in Transformers: Characterizing Changes in Temporal Structure of Attention Scores During Training 9 Feb 2025 · 0 repositories · arXiv:2502.06902
-
EAP-GP: Mitigating Saturation Effect in Gradient-based Automated Circuit Identification 7 Feb 2025 · 0 repositories · arXiv:2502.06852
-
Aligning Human and Machine Attention for Enhanced Supervised Learning 4 Feb 2025 · 0 repositories · arXiv:2502.06811
-
Harmonic Loss Trains Interpretable AI Models 3 Feb 2025 · 1 repository · arXiv:2502.01628
-
Polynomial, trigonometric, and tropical activations 3 Feb 2025 · 1 repository · arXiv:2502.01247
-
LIBRA: Measuring Bias of Large Language Model from a Local Context 2 Feb 2025 · 0 repositories · arXiv:2502.01679
-
Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithms 1 Feb 2025 · 0 repositories · arXiv:2502.00234
-
AlphaAdam:Asynchronous Masked Optimization with Dynamic Alpha for Selective Updates 30 Jan 2025 · 0 repositories · arXiv:2501.18094
-
Structure Development in List-Sorting Transformers 30 Jan 2025 · 0 repositories · arXiv:2501.18666
-
Weight-based Analysis of Detokenization in Language Models: Understanding the First Stage of Inference Without Inference 27 Jan 2025 · 0 repositories · arXiv:2501.15754
-
An Attempt to Unraveling Token Prediction Refinement and Identifying Essential Layers of Large Language Models 25 Jan 2025 · 0 repositories · arXiv:2501.15054
-
FOCUS: First Order Concentrated Updating Scheme 21 Jan 2025 · 0 repositories · arXiv:2501.12243
-
Vision-Language Models for Automated Chest X-ray Interpretation: Leveraging ViT and GPT-2 21 Jan 2025 · 0 repositories · arXiv:2501.12356
-
FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models 18 Jan 2025 · 0 repositories · arXiv:2501.10714
-
Investigating Energy Efficiency and Performance Trade-offs in LLM Inference Across Tasks and DVFS Settings 14 Jan 2025 · 0 repositories · arXiv:2501.08219
-
FinerWeb-10BT: Refining Web Data with LLM-Based Line-Level Filtering 13 Jan 2025 · 1 repository · arXiv:2501.07314
-
Decoding fMRI Data into Captions using Prefix Language Modeling 5 Jan 2025 · 1 repository · arXiv:2501.02570
-
CRRG-CLIP: Automatic Generation of Chest Radiology Reports and Classification of Chest Radiographs 31 Dec 2024 · 1 repository · arXiv:2501.01989
-
Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? Revisiting a Petroglyph 31 Dec 2024 · 0 repositories · arXiv:2501.00659
-
On Adversarial Robustness of Language Models in Transfer Learning 29 Dec 2024 · 0 repositories · arXiv:2501.00066
-
Assessing Text Classification Methods for Cyberbullying Detection on Social Media Platforms 27 Dec 2024 · 0 repositories · arXiv:2412.19928
-
Bridging Auditory Perception and Language Comprehension through MEG-Driven Encoding Models 22 Dec 2024 · 0 repositories · arXiv:2501.03246
-
PsychAdapter: Adapting LLM Transformers to Reflect Traits, Personality and Mental Health 22 Dec 2024 · 1 repository · arXiv:2412.16882
-
No More Adam: Learning Rate Scaling at Initialization is All You Need 16 Dec 2024 · 1 repository · arXiv:2412.11768
-
Priority-Aware Model-Distributed Inference at Edge Networks 16 Dec 2024 · 0 repositories · arXiv:2412.12371
-
GPT-2 Through the Lens of Vector Symbolic Architectures 10 Dec 2024 · 0 repositories · arXiv:2412.07947
-
Superficial Consciousness Hypothesis for Autoregressive Transformers 10 Dec 2024 · 1 repository · arXiv:2412.07278
-
Towards Predictive Communication with Brain-Computer Interfaces integrating Large Language Models 10 Dec 2024 · 0 repositories · arXiv:2412.07355
-
BatchTopK Sparse Autoencoders 9 Dec 2024 · 2 repositories · arXiv:2412.06410Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples)
-
DP-2Stage: Adapting Language Models as Differentially Private Tabular Data Generators 3 Dec 2024 · 1 repository · arXiv:2412.02467
-
Impact of Data Snooping on Deep Learning Models for Locating Vulnerabilities in Lifted Code 3 Dec 2024 · 0 repositories · arXiv:2412.02048
-
The Asymptotic Behavior of Attention in Transformers 3 Dec 2024 · 0 repositories · arXiv:2412.02682
-
Su-RoBERTa: A Semi-supervised Approach to Predicting Suicide Risk through Social Media using Base Language Models 2 Dec 2024 · 0 repositories · arXiv:2412.01353
-
Tokenizing 3D Molecule Structure with Quantized Spherical Coordinates 2 Dec 2024 · 0 repositories · arXiv:2412.01564
-
Streamlining Prediction in Bayesian Deep Learning 27 Nov 2024 · 1 repository · arXiv:2411.18425Syntology official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 0 honoured, 0 violated, 8 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (of 10 harvested samples)
-
CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuning 26 Nov 2024 · 1 repository · arXiv:2411.17426
-
Distributed Sign Momentum with Local Steps for Training Transformers 26 Nov 2024 · 1 repository · arXiv:2411.17866
-
Pretrained LLM Adapted with LoRA as a Decision Transformer for Offline RL in Quantitative Trading 26 Nov 2024 · 1 repository · arXiv:2411.17900
-
Adaptive Circuit Behavior and Generalization in Mechanistic Interpretability 25 Nov 2024 · 0 repositories · arXiv:2411.16105
-
Development of Pre-Trained Transformer-based Models for the Nepali Language 24 Nov 2024 · 0 repositories · arXiv:2411.15734
Tasks archive 2025-07-28
20 shown of 398 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
| Task | Papers |
|---|---|
| Language Modelling | 240 |
| Language Modeling | 183 |
| Text Generation | 128 |
| Sentence | 49 |
| Question Answering | 40 |
| Decoder | 39 |
| Large Language Model | 30 |
| Transfer Learning | 28 |
| GPU | 26 |
| Retrieval | 23 |
| Diversity | 20 |
| Word Embeddings | 20 |
| In-Context Learning | 19 |
| model | 19 |
| Data Augmentation | 18 |
| Text Classification | 17 |
| Translation | 17 |
| Response Generation | 16 |
| Sentiment Analysis | 16 |
| Reinforcement Learning (RL) | 15 |
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections