Methods › Natural Language Processing › Language Models › Chinchilla
Chinchilla
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Chinchilla is a 70B parameters model trained as a compute-optimal model with 1.4 trillion tokens. Findings suggest that these types of models are trained optimally by equally scaling both model size and training tokens. It uses the same compute budget as Gopher but with 4x more training data. Chinchilla and Gopher are trained for the same number of FLOPs. It is trained using MassiveText using a slightly modified SentencePiece tokenizer. More architectural details in the paper.
In Syntology repository known, no verified implementation (declared_code; repositories: 2 listed of 2 in the graph, strongest link: mentioned on GitHub). The archive credits no paper with an arXiv id for this method, so whether this answer (about arXiv:2203.15556) is about this method is not checked.
Origin paper of this answer, arXiv:2203.15556 (its page on this site · its page on Syntology): Syntology's run record for its harvested code, 8 ran (of which 3 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 0 violated, 3 with no contract checked; 3 where Syntology's instrument failed) · 3 unverified (of 11 harvested samples) · 4 pointer-only (licence). The instrument figure counts failures of Syntology's instrument, not of the code. Ran means executed on a synthesized input, not that the code is correct.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 25 · with a code link: 11 · where Syntology ran a sample: 6 (6 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (6 of 25 tagged: 6 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 25 of 25, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
Superposition Yields Robust Neural Scaling 15 May 2025 · 1 repository · arXiv:2505.10465
-
Compute-Optimal LLMs Provably Generalize Better With Scale 21 Apr 2025 · 0 repositories · arXiv:2504.15208
-
Scaling Inference-Efficient Language Models 30 Jan 2025 · 0 repositories · arXiv:2501.18107
-
Physics of Skill Learning 21 Jan 2025 · 1 repository · arXiv:2501.12391
-
The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws 21 Jan 2025 · 0 repositories · arXiv:2501.12486
-
How Much Can We Forget about Data Contamination? 4 Oct 2024 · 1 repository · arXiv:2410.03249Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 2 harvested samples)
-
Energy Estimation of Last Mile Electric Vehicle Routes 21 Aug 2024 · 0 repositories · arXiv:2408.12006
-
Scaling Law with Learning Rate Annealing 20 Aug 2024 · 0 repositories · arXiv:2408.11029
-
Time Matters: Scaling Laws for Any Budget 27 Jun 2024 · 0 repositories · arXiv:2406.18922
-
Reconciling Kaplan and Chinchilla Scaling Laws 12 Jun 2024 · 1 repository · arXiv:2406.12907Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
Revisiting MoE and Dense Speed-Accuracy Comparisons for LLM Training 23 May 2024 · 1 repository · arXiv:2405.15052
-
More Compute Is What You Need 30 Apr 2024 · 0 repositories · arXiv:2404.19484
-
MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies 9 Apr 2024 · 3 repositories · arXiv:2404.06395Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
VBART: The Turkish LLM 2 Mar 2024 · 0 repositories · arXiv:2403.01308
-
A Resource Model For Neural Scaling Law 7 Feb 2024 · 0 repositories · arXiv:2402.05164
-
Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws 31 Dec 2023 · 0 repositories · arXiv:2401.00448
-
The Falcon Series of Open Language Models 28 Nov 2023 · 0 repositories · arXiv:2311.16867
-
Language Modeling Is Compression 19 Sep 2023 · 1 repository · arXiv:2309.10668Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 5 where Syntology's instrument failed) · 2 unverified (of 8 harvested samples)
-
Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla 18 Jul 2023 · 0 repositories · arXiv:2307.09458
-
Tune As You Scale: Hyperparameter Optimization For Compute Efficient Training 13 Jun 2023 · 0 repositories · arXiv:2306.08055
-
Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster 6 Apr 2023 · 2 repositories · arXiv:2304.03208
-
Accelerating Large Language Model Decoding with Speculative Sampling 2 Feb 2023 · 5 repositories · arXiv:2302.01318
-
An Information-Theoretic Analysis of Compute-Optimal Neural Scaling Laws 2 Dec 2022 · 0 repositories · arXiv:2212.01365
-
Galactica: A Large Language Model for Science 16 Nov 2022 · 1 repository · arXiv:2211.09085Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)
-
Training Compute-Optimal Large Language Models 29 Mar 2022 · 2 repositories · arXiv:2203.15556Syntology 8 ran (of which 3 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 0 violated, 3 with no contract checked; 3 where Syntology's instrument failed) · 3 unverified (of 11 harvested samples) · 4 pointer-only (licence)
Tasks archive 2025-07-28
20 shown of 106 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections