Methods › Natural Language Processing › Language Models › Pythia
Pythia
Introduced by Stella Biderman et al. in Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Pythia is a suite of decoder-only autoregressive language models all trained on public data seen in the exact same order and ranging in size from 70M to 12B parameters. The model architecture and hyperparameters largely follow GPT-3, with a few notable deviations based on recent advances in best practices for large scale language modeling.
In Syntology repository known, no verified implementation (declared_code; repositories: 5 listed of 5 in the graph, strongest link: official). The lookup matched a paper by its title, arXiv:2304.01373, not a method entry. This answer is about arXiv:2304.01373, the paper the archive credits with this method.
Origin paper of this answer, arXiv:2304.01373 (its page on this site · its page on Syntology).
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 60 · with a code link: 36 · where Syntology ran a sample: 23 (20 with a run with no instrument failure, 3 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (23 of 60 tagged: 20 with a run with no instrument failure, 3 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 60 of 60, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
LexiMark: Robust Watermarking via Lexical Substitutions to Enhance Membership Verification of an LLM's Textual Training Data 17 Jun 2025 · 1 repository · arXiv:2506.14474
-
What Happens During the Loss Plateau? Understanding Abrupt Learning in Transformers 16 Jun 2025 · 1 repository · arXiv:2506.13688Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
Stochastic Chameleons: Irrelevant Context Hallucinations Reveal Class-Based (Mis)Generalization in LLMs 28 May 2025 · 0 repositories · arXiv:2505.22630
-
Pretraining Language Models to Ponder in Continuous Space 27 May 2025 · 1 repository · arXiv:2505.20674Syntology official (archive's flag): 3 ran · 6 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 5 where Syntology's instrument failed) · 0 unverified (of 6 harvested samples) · 4 pointer-only (licence)
-
Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning 16 May 2025 · 1 repository · arXiv:2505.11004
-
Memorization or Interpolation ? Detecting LLM Memorization through Input Perturbation Analysis 5 May 2025 · 0 repositories · arXiv:2505.03019
-
An Empirical Study of the Role of Incompleteness and Ambiguity in Interactions with Large Language Models 23 Mar 2025 · 0 repositories · arXiv:2503.17936
-
I Predict Therefore I Am: Is Next Token Prediction Enough to Learn Human-Interpretable Concepts from Data? 12 Mar 2025 · 0 repositories · arXiv:2503.08980
-
PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs 12 Mar 2025 · 1 repository · arXiv:2503.09543
-
Interrogating LLM design under a fair learning doctrine 22 Feb 2025 · 0 repositories · arXiv:2502.16290
-
Revisiting Privacy, Utility, and Efficiency Trade-offs when Fine-Tuning Large Language Models 18 Feb 2025 · 0 repositories · arXiv:2502.13313
-
RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models 13 Feb 2025 · 0 repositories · arXiv:2502.09003Syntology 5 ran (of which 5 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; every one of the 5 samples that ran constructed an object rather than computing a result (of 5 harvested samples) · 5 pointer-only (licence)
-
MemHunter: Automated and Verifiable Memorization Detection at Dataset-scale in LLMs 10 Dec 2024 · 0 repositories · arXiv:2412.07261
-
Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuning 21 Nov 2024 · 1 repository · arXiv:2411.14497
-
Explaining and Improving Contrastive Decoding by Extrapolating the Probabilities of a Huge and Hypothetical LM 3 Nov 2024 · 1 repository · arXiv:2411.01610
-
Efficient Training of Sparse Autoencoders for Large Language Models via Layer Groups 28 Oct 2024 · 0 repositories · arXiv:2410.21508
-
Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA 28 Oct 2024 · 0 repositories · arXiv:2410.20672
-
Hallucination Detox: Sensitivity Dropout (SenD) for Large Language Model Training 20 Oct 2024 · 0 repositories · arXiv:2410.15460
-
Tending Towards Stability: Convergence Challenges in Small Language Models 15 Oct 2024 · 1 repository · arXiv:2410.11451
-
Context-Parametric Inversion: Why Instruction Finetuning Can Worsen Context Reliance 14 Oct 2024 · 0 repositories · arXiv:2410.10796
-
Large Language Model Evaluation via Matrix Nuclear-Norm 14 Oct 2024 · 1 repository · arXiv:2410.10672
-
Local and Global Decoding in Text Generation 14 Oct 2024 · 1 repository · arXiv:2410.10810Syntology official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 1 violated, 3 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified (of 12 harvested samples) · 12 pointer-only (licence)
-
Lightweight Deep Learning Framework for Accurate Particle Flow Energy Reconstruction 8 Oct 2024 · 1 repository · arXiv:2410.07250
-
Order of Magnitude Speedups for LLM Membership Inference 22 Sep 2024 · 0 repositories · arXiv:2409.14513
-
Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Data 12 Sep 2024 · 0 repositories · arXiv:2409.11423
-
Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review 10 Sep 2024 · 0 repositories · arXiv:2409.06131
-
Demystifying Verbatim Memorization in Large Language Models 25 Jul 2024 · 1 repository · arXiv:2407.17817Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 0 violated, 3 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 6 harvested samples)
-
Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data 20 Jul 2024 · 0 repositories · arXiv:2407.14985Syntology 11 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 3 honoured, 1 violated, 0 with no contract checked; 7 where Syntology's instrument failed) · 4 unverified (of 15 harvested samples)
-
MathCAMPS: Fine-grained Synthesis of Mathematical Problems From Human Curricula 1 Jul 2024 · 1 repository · arXiv:2407.00900Syntology official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 7 where Syntology's instrument failed) · 0 unverified (of 8 harvested samples)
-
Evaluating n-Gram Novelty of Language Models Using Rusty-DAWG 18 Jun 2024 · 1 repository · arXiv:2406.13069
-
Semantic Membership Inference Attack against Large Language Models 14 Jun 2024 · 0 repositories · arXiv:2406.10218
-
MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models 10 Jun 2024 · 1 repository · arXiv:2406.06046Syntology official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 5 honoured, 2 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified (of 12 harvested samples) · 4 pointer-only (licence)
-
Causal Estimation of Memorisation Profiles 6 Jun 2024 · 1 repository · arXiv:2406.04327Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)
-
How Truncating Weights Improves Reasoning in Language Models 5 Jun 2024 · 0 repositories · arXiv:2406.03068
-
Zyda: A 1.3T Dataset for Open Language Modeling 4 Jun 2024 · 1 repository · arXiv:2406.01981
-
MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series 29 May 2024 · 1 repository · arXiv:2405.19327Syntology official (archive's flag): 12 ran · 12 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 8 where Syntology's instrument failed) · 2 unverified (of 14 harvested samples) · 14 pointer-only (licence)
-
Towards Reliable Latent Knowledge Estimation in LLMs: Zero-Prompt Many-Shot Based Factual Knowledge Extraction 19 Apr 2024 · 1 repository · arXiv:2404.12957
-
Learn Your Reference Model for Real Good Alignment 15 Apr 2024 · 0 repositories · arXiv:2404.09656
-
Mechanistic Understanding and Mitigation of Language Model Non-Factual Hallucinations 27 Mar 2024 · 1 repository · arXiv:2403.18167Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 1 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 5 harvested samples) · 5 pointer-only (licence)
-
The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization 24 Mar 2024 · 1 repository · arXiv:2403.17031Syntology official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 1 honoured, 0 violated, 6 with no contract checked; 1 where Syntology's instrument failed) · 5 unverified (of 13 harvested samples)
-
Pandora's White-Box: Precise Training Data Detection and Extraction in Large Language Models 26 Feb 2024 · 1 repository · arXiv:2402.17012Syntology 7 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified (of 9 harvested samples)
-
CausalGym: Benchmarking causal interpretability methods on linguistic tasks 19 Feb 2024 · 2 repositories · arXiv:2402.12560Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
Model Editing with Canonical Examples 9 Feb 2024 · 1 repository · arXiv:2402.06155Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (of 4 harvested samples) · 4 pointer-only (licence)
-
Successor Heads: Recurring, Interpretable Attention Heads In The Wild 14 Dec 2023 · 0 repositories · arXiv:2312.09230
-
DYAD: A Descriptive Yet Abjuring Density efficient approximation to linear neural network layers 11 Dec 2023 · 1 repository · arXiv:2312.06881Syntology official (archive's flag): 1 ran · 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result (of 1 harvested sample) · 1 pointer-only (licence)
-
SparQ Attention: Bandwidth-Efficient LLM Inference 8 Dec 2023 · 1 repository · arXiv:2312.04985Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 2 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)
-
Scalable Extraction of Training Data from (Production) Language Models 28 Nov 2023 · 0 repositories · arXiv:2311.17035
-
How do Language Models Bind Entities in Context? 26 Oct 2023 · 1 repository · arXiv:2310.17191Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
Characterizing Mechanisms for Factual Recall in Language Models 24 Oct 2023 · 0 repositories · arXiv:2310.15910
-
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning 10 Oct 2023 · 2 repositories · arXiv:2310.06694Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)
-
Efficient Streaming Language Models with Attention Sinks 29 Sep 2023 · 6 repositories · arXiv:2309.17453Syntology official (archive's flag): 6 ran · 8 ran (of which 4 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 4 where Syntology's instrument failed) · 3 unverified (of 11 harvested samples) · 2 pointer-only (licence)
-
TART: A plug-and-play Transformer module for task-agnostic reasoning 21 Sep 2023 · 1 repository
-
Continual Pre-Training of Large Language Models: How to (re)warm your model? 8 Aug 2023 · 2 repositories · arXiv:2308.04014
-
Universal and Transferable Adversarial Attacks on Aligned Language Models 27 Jul 2023 · 25 repositories · arXiv:2307.15043Syntology official: no sample here; runs from other or unrecorded repositories · 39 ran (of which 7 constructed an object rather than computing a result; 19 with no instrument failure: 4 honoured, 1 violated, 14 with no contract checked; 20 where Syntology's instrument failed) · 22 unverified (of 61 harvested samples) · 4 pointer-only (licence)
-
Stay on topic with Classifier-Free Guidance 30 Jun 2023 · 0 repositories · arXiv:2306.17806
-
EaSyGuide : ESG Issue Identification Framework leveraging Abilities of Generative Large Language Models 11 Jun 2023 · 1 repository · arXiv:2306.06662
-
Early Weight Averaging meets High Learning Rates for LLM Pre-training 5 Jun 2023 · 1 repository · arXiv:2306.03241Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
Emergent inabilities? Inverse scaling over the course of pretraining 24 May 2023 · 0 repositories · arXiv:2305.14681
-
Emergent and Predictable Memorization in Large Language Models 21 Apr 2023 · 2 repositories · arXiv:2304.11158Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling 3 Apr 2023 · 4 repositories · arXiv:2304.01373
Tasks archive 2025-07-28
20 shown of 58 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections