Papers › Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

3 Apr 2023arXiv:2304.01373archive 2025-07-28

Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, Oskar van der Wal

How do large language models (LLMs) develop and evolve over the course of training? How do these patterns change as models scale? To answer these questions, we introduce Pythia, a suite of 16 LLMs all trained on public data seen in the exact same order and ranging in size from 70M to 12B parameters. We provide public access to 154 checkpoints for each one of the 16 models, alongside tools to download and reconstruct their exact training dataloaders for further study. We intend Pythia to facilitate research in many areas, and we present several case studies including novel results in memorization, term frequency effects on few-shot performance, and reducing gender bias. We demonstrate that this highly controlled setup can be used to yield novel insights toward LLMs and their training dynamics. Trained models, analysis code, training code, and training data can be found at https://github.com/EleutherAI/pythia.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

eleutherai/gpt-neox officialmentioned in papermentioned on GitHubpytorchApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to PSLM: Parallel Generation of Text and Speech with LLMs for Low-Latency Spoken Dialogue Systems (arXiv:2406.12428; that paper's own run record: 6 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 6 where Syntology's instrument failed) · 2 unverified); nothing checks that they implement this paper's method. report
eleutherai/pythia officialmentioned in papermentioned on GitHubpytorchApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Emergent and Predictable Memorization in Large Language Models (arXiv:2304.11158; that paper's own run record: 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified) and arXiv:2507.00322 (its page on Syntology; that paper's own run record: 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
Lightning-AI/lit-gpt mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Llama 2: Open Foundation and Fine-Tuned Chat Models (arXiv:2307.09288; that paper's own run record: 33 ran (of which 7 constructed an object rather than computing a result; 20 with no instrument failure: 1 honoured, 1 violated, 18 with no contract checked; 13 where Syntology's instrument failed) · 19 unverified); nothing checks that they implement this paper's method. report
jzhang38/tinyllama mentioned on GitHubpytorchApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to TinyLlama: An Open-Source Small Language Model (arXiv:2401.02385; that paper's own run record: 8 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 2 honoured, 2 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 1 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Common Sense ReasoningCoreference ResolutionLanguage ModellingMemorizationQuestion Answering

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Common Sense Reasoning ARC (Challenge) Pythia 12B (5-shot) Accuracy 36.8 #49 of 54 Archive leaderboard report
Common Sense Reasoning ARC (Challenge) Pythia 12B (0-shot) Accuracy 31.8 #51 of 54 Archive leaderboard report
Common Sense Reasoning ARC (Easy) Pythia 12B (5-shot) Accuracy 71.5 #28 of 47 Archive leaderboard report
Common Sense Reasoning ARC (Easy) Pythia 12B (0-shot) Accuracy 70.2 #33 of 47 Archive leaderboard report
Common Sense Reasoning WinoGrande Pythia 12B (5-shot) Accuracy 66.6 #43 of 77 Archive leaderboard report
Common Sense Reasoning WinoGrande Pythia 12B (0-shot) Accuracy 63.9 #47 of 77 Archive leaderboard report
Common Sense Reasoning WinoGrande Pythia 6.9B (0-shot) Accuracy 60.9 #49 of 77 Archive leaderboard report
Common Sense Reasoning WinoGrande Pythia 2.8B (0-shot) Accuracy 59.4 #52 of 77 Archive leaderboard report
Coreference Resolution Winograd Schema Challenge Pythia 12B (0-shot) Accuracy 54.8 #71 of 82 Archive leaderboard report
Coreference Resolution Winograd Schema Challenge Pythia 2.8B (0-shot) Accuracy 38.5 #79 of 82 Archive leaderboard report
Coreference Resolution Winograd Schema Challenge Pythia 6.9B (0-shot) Accuracy 36.5 #81 of 82 Archive leaderboard report
Coreference Resolution Winograd Schema Challenge Pythia 12B (5-shot) Accuracy 36.5 #82 of 82 Archive leaderboard report
Language Modelling LAMBADA Pythia 12B (0-shot) Accuracy 70.46 #22 of 37 Archive leaderboard report
Language Modelling LAMBADA Pythia 6.9B (0-shot) Accuracy 67.28 #26 of 37 Archive leaderboard report
Language Modelling LAMBADA Pythia 12B(Zero-Shot) Perplexity 3.92 #35 of 37 Archive leaderboard report
Language Modelling LAMBADA Pythia 6.9B(Zero-Shot) Perplexity 4.45 #36 of 37 Archive leaderboard report
Question Answering PIQA Pythia 12B (5-shot) Accuracy 76.7 #44 of 67 Archive leaderboard report
Question Answering PIQA Pythia 12B (0-shot) Accuracy 76 #46 of 67 Archive leaderboard report
Question Answering PIQA Pythia 6.9B (0-shot) Accuracy 75.2 #49 of 67 Archive leaderboard report
Question Answering PIQA Pythia 1B (5-shot) Accuracy 70.4 #57 of 67 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Introduced by this paper: Pythia

Pythia

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections