Datasets › MATH

MATH

Introduced by Dan Hendrycks et al. in Measuring Mathematical Problem Solving With the MATH Dataset5 Mar 2021 archive 2025-07-28

MATH is a new dataset of 12,500 challenging competition mathematics problems. Each problem in MATH has a full step-by-step solution which can be used to teach models to generate answer derivations and explanations.

Source: Hendrycks et al.

Image source: Hendrycks et al.

Benchmarks archive 2025-07-28

All 2 leaderboards whose dataset resolves to this page shown (sort by any header). "First row" is the archive's own first row at snapshot, in the archive's row order; nothing here re-ranks and metric direction is not asserted.

Syntology 54 newer papers were judged to report on this dataset's benchmarks so far (Claude Sonnet 4.5, reading their tables: a model's judgement, not a result; 0 placed): Math Word Problem Solving · MATH (53 judged, 0 placed), Math Word Problem Solving · MATH minival (1 judged, 0 placed). Drawn from 8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read).

First row (archive order)PaperCode
Math Word Problem Solving MATH Gemini 2.0 Flash Experimental Accuracy 89.7 — — 135 Compare
Math Word Problem Solving MATH minival Process Supervision (GPT-4) Accuracy 78.2 Let's Verify Step by Step openai/prm800k +2 1 Compare

Papers archive 2025-07-28

papers with a benchmark row: 37 · with a code link: 34 · where Syntology ran a sample: 21 (17 with a run with no instrument failure, 4 where every run was a failure of Syntology's instrument) Syntology

Show: all papers with a benchmark rowonly where code ran (21 of 37 with a benchmark row: 17 with a run with no instrument failure, 4 where every run was a failure of Syntology's instrument)

Page 1 of 1: papers 1 to 37 of 37 with a leaderboard row on this dataset's benchmarks, newest first by the archive's date (ties by slug; undated papers last).

The archive's own "papers using this dataset" list was never published, so this is the benchmark-backed subset, not that list; the archive's count for this dataset is 1,330. The Syntology column is from Syntology's graph, stated per sample; it is not part of any archive number. A line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified” (C is a part of N, never taken away from it); the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. When the archive marks a repository official for the paper, the cell starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record.

PaperCodeResultsDateSamples run Syntology
OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data 1 4 2 Oct 2024 12 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 7 where Syntology's instrument failed) · 3 unverified
Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement 0 6 18 Sep 2024 not harvested
Qwen2 Technical Report 6 1 15 Jul 2024 not harvested
Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs 1 1 26 Jun 2024 official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified (12 pointer-only for licence)
DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving 1 8 18 Jun 2024 official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 8 where Syntology's instrument failed) · 2 unverified
AlphaMath Almost Zero: Process Supervision without Process 1 1 6 May 2024 not harvested
Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing 1 1 18 Apr 2024 official (archive's flag): 14 ran · 14 ran (of which 1 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 1 violated, 3 with no contract checked; 9 where Syntology's instrument failed) · 4 unverified (18 pointer-only for licence)
MACM: Utilizing a Multi-Agent System for Condition Mining in Solving Complex Mathematical Problems 1 1 6 Apr 2024 official: harvested, nothing ran · 0 ran · 1 unverified (1 pointer-only for licence)
Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM 1 1 12 Mar 2024 not harvested
Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoning 0 4 4 Mar 2024 not harvested
An Empirical Study of Data Ability Boundary in LLMs' Math Reasoning 1 4 23 Feb 2024 official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified (11 pointer-only for licence)
OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset 1 12 15 Feb 2024 not harvested
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models 5 2 5 Feb 2024 official (archive's flag): 8 ran · 14 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 11 where Syntology's instrument failed) · 10 unverified (3 pointer-only for licence)
Augmenting Math Word Problems via Iterative Question Composing 1 1 17 Jan 2024 official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (4 pointer-only for licence)
Mixtral of Experts 6 2 8 Jan 2024 5 ran (of which 5 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; every one of the 5 samples that ran constructed an object rather than computing a result
Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks 2 2 5 Jan 2024 not harvested
Gemini: A Family of Highly Capable Multimodal Models 1 2 19 Dec 2023 not harvested
Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations 3 3 14 Dec 2023 not harvested
Mistral 7B 6 1 10 Oct 2023 official (archive's flag): 9 ran · 10 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (2 pointer-only for licence)
MuggleMath: Assessing the Impact of Query and Response Augmentation on Math Reasoning 1 3 9 Oct 2023 not harvested
MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning 1 6 5 Oct 2023 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified
ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving 1 8 29 Sep 2023 official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified (9 pointer-only for licence)
MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models 1 3 21 Sep 2023 official (archive's flag): 15 ran · 15 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 14 where Syntology's instrument failed) · 7 unverified
OpenChat: Advancing Open-source Language Models with Mixed-Quality Data 1 2 20 Sep 2023 official: harvested, nothing ran · 0 ran · 1 unverified
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct 1 4 18 Aug 2023 10 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 3 where Syntology's instrument failed) · 6 unverified (16 pointer-only for licence)
Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification 1 5 15 Aug 2023 not harvested
Cumulative Reasoning with Large Language Models 1 2 8 Aug 2023 official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (6 pointer-only for licence)
Skills-in-Context Prompting: Unlocking Compositionality in Large Language Models 0 1 1 Aug 2023 not harvested
Let's Verify Step by Step 3 1 31 May 2023 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 2 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified
PaLM 2 Technical Report 1 2 17 May 2023 not harvested
Progressive-Hint Prompting Improves Reasoning in Large Language Models 1 1 19 Apr 2023 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified (4 pointer-only for licence)
Sparks of Artificial General Intelligence: Early experiments with GPT-4 3 1 22 Mar 2023 9 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 0 honoured, 0 violated, 9 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified
LLaMA: Open and Efficient Foundation Language Models 57 8 27 Feb 2023 official: no sample here; runs from other or unrecorded repositories · 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified (4 pointer-only for licence)
PAL: Program-aided Language Models 3 1 18 Nov 2022 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified
Galactica: A Large Language Model for Science 1 7 16 Nov 2022 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified
Solving Quantitative Reasoning Problems with Language Models 1 13 29 Jun 2022 not harvested
Measuring Mathematical Problem Solving With the MATH Dataset 5 8 5 Mar 2021 official (archive's flag): 2 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (1 pointer-only for licence)

Dataset loaders archive 2025-07-28

2 loaders as listed in the archive; links are outbound and not re-checked here.

Tasks archive 2025-07-28

License archive 2025-07-28

MIT

Modalities archive 2025-07-28

Languages archive 2025-07-28

Variants archive 2025-07-28

  • MATH
  • MATH minival

2 variant names, as the archive lists them.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections