</> Code with Papers by Syntology

Browse › Reasoning › Math Word Problem Solving › MATH

Math Word Problem Solving archive 2025-07-28

MATH Benchmark (Math Word Problem Solving)

135 rows 121 with code listed 2 metrics Dataset page

A math word problem is a mathematical exercise (such as in a textbook, worksheet, or exam) where significant background information on the problem is presented in ordinary language rather than in mathematical notation. As most word problems involve a narrative of some sort, they are sometimes referred to as story problems and may vary in the amount of technical language used.

The archive carries no text for this table; the description above is the archive's text for the task Math Word Problem Solving. archive 2025-07-28

Over time archive 2025-07-28

The chart needs JavaScript; the table below carries every value.

Direction inferred from the metric name, not from the archive: Accuracy (higher is better). Not inferred (points only, no best-so-far line): Parameters (Billions). Points are placed at the row's paper date; 132 of 135 rows carry one.

Results archive 2025-07-28

Archive rows end at the archive snapshot, 2025-07-28: no result published after that date is in this table. Rank is the archive's row order at that snapshot; not re-ranked here. Metric values are the archive's strings. Column headers sort the table in your browser; each row keeps its archive rank.

Paper Code Ran Syntology Report
1 Gemini 2.0 Flash Experimental 89.7 – – – not matched report
2 Qwen2.5-Math-72B-Instruct(TIR,Greedy) 88.172 ✓ Paper – 2024 no code linked report
3 GPT-4 Turbo (MACM, w/code, voting) 87.920 – Paper Code 2024 0 of 1 ran · 1 unverified report
4 Qwen2.5-Math-72B-Instruct(COT,Greedy) 85.972 ✓ Paper – 2024 no code linked report
5 Qwen2.5-Math-7B-Instruct(TIR,Greedy) 85.27 ✓ Paper – 2024 no code linked report
6 GPT-4-code model (CSV, w/ code, SC, k=16) 84.3 – Paper Code 2023 linked, not harvested report
7 Qwen2-Math-72B-Instruct(greedy) 84.072 ✓ Paper Code 2024 linked, not harvested report
8 Qwen2.5-Math-7B-Instruct(COT,Greedy) 83.67 ✓ Paper – 2024 no code linked report
9 Qwen2.5-Math-1.5B-Instruct(TIR,Greedy) 79.91.5 ✓ Paper – 2024 no code linked report
10 OpenMath2-Llama3.1-70B (majority@256) 79.6 ✓ Paper Code 2024 0 of 15 ran · 15 unverified report
11 OpenMath2-Llama3.1-8B (majority@256) 76.1 ✓ Paper Code 2024 0 of 15 ran · 15 unverified report
12 Qwen2.5-Math-1.5B-Instruct(COT,Greedy) 75.81.5 ✓ Paper – 2024 no code linked report
13 GPT-4-code model (CSV, w/ code) 73.5 – Paper Code 2023 linked, not harvested report
14 CR (GPT-4-turbo model, w/ code) 72.2 – Paper Code 2023 5 of 6 ran · 1 unverified report
15 OpenMath2-Llama3.1-70B 71.9 ✓ Paper Code 2024 0 of 15 ran · 15 unverified report
16 LogicNet (with code interpreter) 71.2 ✓ Paper Code 2023 linked, not harvested report
17 Qwen2-72B-Instruct-Step-DPO (0-shot CoT, w/o code) 70.8 ✓ Paper Code 2024 7 of 12 ran · 5 unverified report
18 GPT-4-code model (w/ code) 69.7 – Paper Code 2023 linked, not harvested report
19 OpenMath2-Llama3.1-8B 67.8 ✓ Paper Code 2024 0 of 15 ran · 15 unverified report
20 AlphaMath-7B-SBS@3 66.3 – Paper Code 2024 linked, not harvested report
21 Minerva 62B (maj5@256) 64.962 – Paper Code 2022 linked, not harvested report
22 DAMOMath-7B 64.57 ✓ – – not matched report
23 MMOS-DeepSeekMath-7B(0-shot,k=50) 63.77 ✓ Paper Code 2024 9 of 11 ran · 2 unverified report
24 GPT-4-code model (w/o code) 60.8 – Paper Code 2023 linked, not harvested report
25 OpenMath-CodeLlama-70B (w/ code, SC, k=50) 60.470 ✓ Paper Code 2024 linked, not harvested report
26 OpenMath-CodeLlama-34B (w/ code, SC, k=50) 60.234 ✓ Paper Code 2024 linked, not harvested report
27 ToRA-Code 34B model (w/ code, SC, k=50) 60.034 ✓ Paper Code 2023 5 of 12 ran · 7 unverified report
28 DeepSeekMATH-RL-7B (w/ code, greedy decoding) 58.87 ✓ Paper Code 2024 8 of 24 ran · 16 unverified report
29 OpenMath-Llama2-70B (w/ code, SC, k=50) 58.370 ✓ Paper Code 2024 linked, not harvested report
30 CR (GPT-4 model, w/o code) 58.0 – Paper Code 2023 5 of 6 ran · 1 unverified report
31 OpenMath-CodeLlama-13B (w/ code, SC, k=50) 57.613 ✓ Paper Code 2024 linked, not harvested report
32 OpenMath-Mistral-7B (w/ code, SC, k=50) 57.27 ✓ Paper Code 2024 linked, not harvested report
33 ToRA 70B (w/ code, SC, k=50) 56.970 ✓ Paper Code 2023 5 of 12 ran · 7 unverified report
34 SKiC (GPT-4 model) 56.4 – Paper – 2023 no code linked report
35 DART-Math-Llama3-70B-Prop2Diff (0-shot CoT, w/o code) 56.170 ✓ Paper Code 2024 10 of 12 ran · 2 unverified report
36 OpenMath-CodeLlama-7B (w/ code, SC, k=50) 55.67 ✓ Paper Code 2024 linked, not harvested report
37 MMOS-DeepSeekMath-7B(0-shot) 55.07 ✓ Paper Code 2024 9 of 11 ran · 2 unverified report
38 DART-Math-Llama3-70B-Uniform (0-shot CoT, w/o code) 54.970 ✓ Paper Code 2024 10 of 12 ran · 2 unverified report
39 PHP (GPT-4 model) 53.9 – Paper Code 2023 3 of 4 ran · 1 unverified report
40 DART-Math-DSMath-7B-Prop2Diff (0-shot CoT, w/o code) 53.67 ✓ Paper Code 2024 10 of 12 ran · 2 unverified report
41 Gemini Ultra (4-shot) 53.2 – Paper Code 2023 linked, not harvested report
42 DART-Math-DSMath-7B-Uniform (0-shot CoT, w/o code) 52.97 ✓ Paper Code 2024 10 of 12 ran · 2 unverified report
43 GPT-4 model (w/ code, PAL) 51.8 – Paper Code 2022 3 of 3 ran · 0 unverified report
44 DeepSeekMATH-RL-7B (greedy decoding) 51.77 ✓ Paper Code 2024 8 of 24 ran · 16 unverified report
45 AlphaLLM (with MCTS) 51 – Paper Code 2024 13 of 18 ran · 5 unverified report
46 ToRA-Code 34B (w/ code) 50.834 ✓ Paper Code 2023 5 of 12 ran · 7 unverified report
47 OpenMath-CodeLlama-70B (w/ code) 50.770 ✓ Paper Code 2024 linked, not harvested report
48 Minerva 540B (maj1@k, k=64) 50.3 – Paper Code 2022 linked, not harvested report
49 ToRA 70B (w/ code) 49.770 ✓ Paper Code 2023 5 of 12 ran · 7 unverified report
50 MMOS-CODE-34B(0-shot) 49.534 ✓ Paper Code 2024 9 of 11 ran · 2 unverified report
51 DeepSeekMath-7B-KPMath-Plus 48.87 – Paper – 2024 no code linked report
52 PaLM 2 (few-shot, k=4, SC) 48.8 – Paper Code 2023 linked, not harvested report
53 Llemma-34B-KPMath-Plus 48.634 – Paper – 2024 no code linked report
54 OpenMath-CodeLlama-34B (w/ code) 48.334 ✓ Paper Code 2024 linked, not harvested report
55 Shepherd + DeepSeek-67B (SFT on MetaMATH + PRM rerank, k=256) 48.167 ✓ Paper Code 2023 linked, not harvested report
56 ToRA-Code 13B (w/ code) 48.113 ✓ Paper Code 2023 5 of 12 ran · 7 unverified report
57 Minerva 8B (maj5@256) 47.68 – Paper Code 2022 linked, not harvested report
58 Mistral-7B-KPMath-Plus 46.87 ✓ Paper – 2024 no code linked report
59 DART-Math-Llama3-8B-Prop2Diff (0-shot CoT, w/o code) 46.68 ✓ Paper Code 2024 10 of 12 ran · 2 unverified report
60 OpenMath-Llama2-70B (w/ code) 46.370 ✓ Paper Code 2024 linked, not harvested report
61 OpenMath-CodeLlama-13B (w/ code) 45.513 ✓ Paper Code 2024 linked, not harvested report
62 DART-Math-Mistral-7B-Prop2Diff (0-shot CoT, w/o code) 45.57 ✓ Paper Code 2024 10 of 12 ran · 2 unverified report
63 DART-Math-Llama3-8B-Uniform (0-shot CoT, w/o code) 45.38 ✓ Paper Code 2024 10 of 12 ran · 2 unverified report
64 MathCoder-CL-34B 45.234 ✓ Paper Code 2023 2 of 2 ran · 0 unverified report
65 MathCoder-L-34B 45.134 ✓ Paper Code 2023 2 of 2 ran · 0 unverified report
66 MMIQC-72B 45.072 ✓ Paper Code 2024 4 of 4 ran · 0 unverified report
67 ToRA-Code 7B (w/ code) 44.67 ✓ Paper Code 2023 5 of 12 ran · 7 unverified report
68 OpenMath-Mistral-7B (w/ code) 44.57 ✓ Paper Code 2024 linked, not harvested report
69 MMOS-CODE-7B(0-shot) 44.37 ✓ Paper Code 2024 9 of 11 ran · 2 unverified report
70 OpenMath-CodeLlama-7B (w/ code) 43.67 ✓ Paper Code 2024 linked, not harvested report
71 Shepherd+Mistral-7B (SFT on MetaMATH + PRM RL+ PRM rerank, k=256) 43.57 ✓ Paper Code 2023 linked, not harvested report
72 DART-Math-Mistral-7B-Uniform (0-shot CoT, w/o code) 43.57 ✓ Paper Code 2024 10 of 12 ran · 2 unverified report
73 Minerva 62B (maj1@k, k=64) 43.462 – Paper Code 2022 linked, not harvested report
74 ToRA 13B (w/ code) 43.013 ✓ Paper Code 2023 5 of 12 ran · 7 unverified report
75 GPT-4 42.5 – Paper Code 2023 0 of 10 ran · 10 unverified report
76 SFT-Mistral-7B 41.87 ✓ – – not matched report
77 Llama2-13B-KPMath-Plus 4113 – Paper – 2024 no code linked report
78 ToRA 7B (w/ code) 40.17 ✓ Paper Code 2023 5 of 12 ran · 7 unverified report
79 MathCoder-CL-13B 35.913 ✓ Paper Code 2023 2 of 2 ran · 0 unverified report
80 MuggleMATH-70B 35.670 ✓ Paper Code 2023 linked, not harvested report
81 PaLM 2 (few-shot, k=4, CoT) 34.3 – Paper Code 2023 linked, not harvested report
82 Minerva 540B 33.6540 – Paper Code 2022 linked, not harvested report
83 Minerva 540B (5-shot) mCoT 33.6540 – Paper Code 2022 0 of 2 ran · 2 unverified report
84 Shepherd + Mistral-7B (SFT on MetaMATH + PRM RL) 33.07 ✓ Paper Code 2023 linked, not harvested report
85 WizardMath-7B-V1.1 33.07 ✓ Paper Code 2023 10 of 16 ran · 6 unverified report
86 Gemini Pro (4-shot) 32.6 – Paper Code 2023 linked, not harvested report
87 MuggleMATH-13B 30.713 ✓ Paper Code 2023 linked, not harvested report
88 MathCoder-CL-7B 30.27 ✓ Paper Code 2023 2 of 2 ran · 0 unverified report
89 MathCoder-L-13B 29.913 ✓ Paper Code 2023 2 of 2 ran · 0 unverified report
90 Qwen2idae-16x14B (4-shot) 29.9 – Paper Code 2024 linked, not harvested report
91 OpenChat-3.5-1210 7B 28.97 – Paper Code 2023 0 of 1 ran · 1 unverified report
92 OpenChat-3.5 7B 28.67 – Paper Code 2023 0 of 1 ran · 1 unverified report
93 Mixtral 8x7B (maj@4) 28.4 – Paper Code 2024 5 of 5 ran · 0 unverified report
94 Minerva 62B (4-shot) 27.662 – Paper Code 2022 linked, not harvested report
95 MetaMath 70B 26.070 ✓ Paper Code 2023 15 of 22 ran · 7 unverified report
96 MuggleMATH 7B 25.87 ✓ Paper Code 2023 linked, not harvested report
97 Minerva 8B (maj1@k, k=64) 25.48 – Paper Code 2022 linked, not harvested report
98 MathCoder-L-7B 23.37 ✓ Paper Code 2023 2 of 2 ran · 0 unverified report
99 WizardMath-70B-V1.0 22.770 ✓ Paper Code 2023 10 of 16 ran · 6 unverified report
100 Camelidae-8×34B (4-shot) 22.6 – Paper Code 2024 linked, not harvested report
101 MetaMath 13B 22.513 ✓ Paper Code 2023 15 of 22 ran · 7 unverified report
102 LLaMA 65B (maj1@k) 20.565 – Paper Code 2023 26 of 58 ran · 32 unverified report
103 GAL 120B (5-shot) mCoT 20.4120 – Paper Code 2022 0 of 2 ran · 2 unverified report
104 MetaMath 7B 19.47 ✓ Paper Code 2023 15 of 22 ran · 7 unverified report
105 davinci-002 175B 19.1175 – Paper Code 2022 linked, not harvested report
106 Branch-Train-MiX 4x7B (sampling top-2 experts) 17.8 – Paper Code 2024 linked, not harvested report
107 GAL 120B <work> 16.6120 – Paper Code 2022 0 of 2 ran · 2 unverified report
108 LLaMA 33B-maj1@k 15.233 – Paper Code 2023 26 of 58 ran · 32 unverified report
109 Minerva 8B 14.18 – Paper Code 2022 linked, not harvested report
110 WizardMath-13B-V1.0 14.013 ✓ Paper Code 2023 10 of 16 ran · 6 unverified report
111 Mistral 7B (maj@4) 13.17 – Paper Code 2023 9 of 11 ran · 2 unverified report
112 GAL 30B (5-shot) mCoT 12.730 – Paper Code 2022 0 of 2 ran · 2 unverified report
113 Mistral 7B (maj@4) 12.77 – Paper Code 2024 5 of 5 ran · 0 unverified report
114 GAL 30B <work> 11.430 – Paper Code 2022 0 of 2 ran · 2 unverified report
115 WizardMath-7B-V1.0 10.77 ✓ Paper Code 2023 10 of 16 ran · 6 unverified report
116 LLaMA 65B 10.665 – Paper Code 2023 26 of 58 ran · 32 unverified report
117 PaLM 540B 8.8540 – Paper Code 2022 linked, not harvested report
118 PaLM 540B (5-shot) mCoT 8.8540 – Paper Code 2022 0 of 2 ran · 2 unverified report
119 LLaMA 13B-maj1@k 8.813 – Paper Code 2023 26 of 58 ran · 32 unverified report
120 LLaMA 33B 7.133 – Paper Code 2023 26 of 58 ran · 32 unverified report
121 LLaMA 7B-maj1@k 6.97 – Paper Code 2023 26 of 58 ran · 32 unverified report
122 GPT-2 (1.5B) 6.91.5 – Paper Code 2021 3 of 3 ran · 0 unverified report
123 GPT-2 (0.7B) 6.40.7 – Paper Code 2021 3 of 3 ran · 0 unverified report
124 GPT-2 (0.3B) 6.20.3 – Paper Code 2021 3 of 3 ran · 0 unverified report
125 GPT-3 13B 5.613 – Paper Code 2021 3 of 3 ran · 0 unverified report
126 PaLM 8B (fine-tuned) 5.68 – Paper Code 2022 linked, not harvested report
127 GPT-2 (0.1B) 5.40.1 – Paper Code 2021 3 of 3 ran · 0 unverified report
128 GPT-3-175B (few-shot) 5.2175 – Paper Code 2021 3 of 3 ran · 0 unverified report
129 GPT-3 175B (8-shot) 5.2175 – Paper Code 2022 0 of 2 ran · 2 unverified report
130 PaLM 62B 4.462 – Paper Code 2022 linked, not harvested report
131 LLaMA 13B 3.913 – Paper Code 2023 26 of 58 ran · 32 unverified report
132 GPT-3-13B (few-shot) 3.013 – Paper Code 2021 3 of 3 ran · 0 unverified report
133 LLaMA 7B 2.97 – Paper Code 2023 26 of 58 ran · 32 unverified report
134 GPT-3 2.7B 2.92.7 – Paper Code 2021 3 of 3 ran · 0 unverified report
135 PaLM 8B 1.58 – Paper Code 2022 linked, not harvested report

All 135 rows shown. 132 link to a paper page on this site; 70 are marked as using additional training data in the archive. No GitHub stars are tracked; "Code" is the first repository the archive lists for the row. The archive carries no row tags, review links or community-submitted rows for this table; none are shown. archive 2025-07-28

Syntology Ran reads "N of M ran · U unverified": of the M code samples Syntology harvested from repositories linked to that row's paper (joined by arXiv id), N executed on a synthesized input and the other U = M−N are unverified (harvested, no recorded run). It counts code from repositories linked to that row's paper, not this result: the row's number was not reproduced and nothing here is a correctness claim. The other cell texts mean no graph line for the row: "linked, not harvested" (the archive links code, Syntology has not harvested it), "no code linked" (no code link in the archive), "not matched" (the row's paper URL matched no paper on this site). 76 rows have a graph line, from 22 distinct papers; 61 rows (17 papers) have at least one sample that ran. Counting each paper once: Syntology ran 137 of 252 samples; 115 unverified. Separately, 78 of those 252 are pointer-only (licence): the site points at that code rather than redistributing it, a licence property recorded for ran and unverified samples alike; each cell's tooltip carries the row's own pointer-only count. Read from the graph 2026-09-24. Per-sample status is on the paper page.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections