Browse › Reasoning › Arithmetic Reasoning › GSM8K

Arithmetic Reasoning archive 2025-07-28

GSM8K Benchmark (Arithmetic Reasoning)

164 rows 118 with code listed 2 metrics Dataset page

Syntology Since the archive (Syntology): none placed; 168 newer papers judged to report on this board so far.

Over time archive 2025-07-28

Direction inferred from the metric name, not from the archive: Accuracy (higher is better). Not inferred (points only, no best-so-far line): Parameters (Billion). Points are placed at the row's paper date, not an evaluation date; 150 of 164 rows carry one.

The chart needs JavaScript; the table below carries every value.

Results archive 2025-07-28

Archive rows end at the archive snapshot, 2025-07-28: no result published after that date is in this table. Rank is the archive's row order at that snapshot; not re-ranked here. Metric values are the archive's strings. Column headers sort the table in your browser; each row keeps its archive rank.

Paper Code Ran Syntology Report
1 Claude 3.5 Sonnet (HPT) 97.72 – Paper Code 2024 linked, not harvested report
2 DUP prompt upon GPT-4 97.1 – Paper Code 2024 linked, not harvested report
3 Qwen2-Math-72B-Instruct (greedy) 96.772 ✓ Paper Code 2024 linked, not harvested report
4 SFT-Mistral-7B (Metamath, OVM, Smart Ensemble) 96.47 ✓ – – not matched report
5 OpenMath2-Llama3.1-70B (majority@256) 96.0 ✓ Paper Code 2024 12 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 7 where Syntology's instrument failed) · 3 unverified report
6 Jiutian-大模型 95.275 – – – not matched report
7 DAMOMath-7B(MetaMath, OVM, BS, Ensemble) 95.17 ✓ – – not matched report
8 Claude 3 Opus (0-shot chain-of-thought) 95 – Paper – 2024 no code linked report
9 OpenMath2-Llama3.1-70B 94.9 ✓ Paper Code 2024 12 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 7 where Syntology's instrument failed) · 3 unverified report
10 GPT-4 (Teaching-Inspired) 94.8 – Paper Code 2024 official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 2 violated, 0 with no contract checked; 5 where Syntology's instrument failed) · 0 unverified report
11 SFT-Mistral-7B (Metamath + ovm +ensemble) 94.137 ✓ – – not matched report
12 OpenMath2-Llama3.1-8B (majority@256) 94.1 ✓ Paper Code 2024 12 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 7 where Syntology's instrument failed) · 3 unverified report
13 Qwen2-72B-Instruct-Step-DPO (0-shot CoT) 94.0 ✓ Paper Code 2024 official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified report
14 DAMOMath-7B(MetaMath, OVM, Ensemble) 93.27 ✓ – – not matched report
15 Claude 3 Sonnet (0-shot chain-of-thought) 92.3 – Paper – 2024 no code linked report
16 AlphaLLM (with MCTS) 9270 – Paper Code 2024 official (archive's flag): 14 ran · 14 ran (of which 1 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 1 violated, 3 with no contract checked; 9 where Syntology's instrument failed) · 4 unverified report
17 OpenMath2-Llama3.1-8B 91.7 ✓ Paper Code 2024 12 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 7 where Syntology's instrument failed) · 3 unverified report
18 PaLM 2 (few-shot, k=8, SC) 91.0 – Paper Code 2023 linked, not harvested report
19 GaC(Qwen2-72B-Instruct + Llama-3-70B-Instruct) 90.91 – Paper Code 2024 official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified report
20 OpenMath-CodeLlama-70B (w/ code, SC, k=50) 90.870 ✓ Paper Code 2024 linked, not harvested report
21 DART-Math-Llama3-70B-Uniform (0-shot CoT, w/o code) 90.470 ✓ Paper Code 2024 official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 8 where Syntology's instrument failed) · 2 unverified report
22 OpenMath-Llama2-70B (w/ code, SC, k=50) 90.170 ✓ Paper Code 2024 linked, not harvested report
23 DART-Math-Llama3-70B-Prop2Diff (0-shot CoT, w/o code) 89.670 ✓ Paper Code 2024 official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 8 where Syntology's instrument failed) · 2 unverified report
24 Shepherd+Mistral-7B (SFT on MetaMATH + PRM RL+ PRM rerank, k=256) 89.17 ✓ Paper Code 2023 linked, not harvested report
25 Llama SFT (Metamath ToRA Ensemble) 89.013 ✓ – – not matched report
26 Minerva 62B (maj5@100) 8962 – Paper Code 2022 linked, not harvested report
27 Claude 3 Haiku (0-shot chain-of-thought) 88.9 – Paper – 2024 no code linked report
28 ToRA-70B (SC, k=50) 88.370 ✓ Paper Code 2023 official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified report
29 DeepSeekMATH-RL-7B 88.27 ✓ Paper Code 2024 official (archive's flag): 8 ran · 14 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 11 where Syntology's instrument failed) · 10 unverified report
30 DART-Math-DSMath-7B-Uniform (0-shot CoT, w/o code) 88.27 ✓ Paper Code 2024 official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 8 where Syntology's instrument failed) · 2 unverified report
31 OpenMath-CodeLlama-34B (w/ code, SC, k=50) 88.034 ✓ Paper Code 2024 linked, not harvested report
32 Claude 2 (0-shot chain-of-thought) 88 – Paper – 2023 no code linked report
33 Shivaay-4B (8-shot chain-of-thought) 87.414 – – – not matched report
34 DeepMind 70B Model (SFT+ORM-RL, ORM reranking) 87.370 ✓ Paper – 2022 no code linked report
35 MMOS-DeepSeekMath-7B(0-shot,k=50) 87.27 ✓ Paper Code 2024 official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified report
36 DeepMind 70B Model (SFT+PRM-RL, PRM reranking) 87.170 ✓ Paper – 2022 no code linked report
37 GPT-4 87.1 – Paper Code 2023 9 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 0 honoured, 0 violated, 9 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified report
38 OpenMath-Mistral-7B (w/ code, SC, k=50) 86.97 ✓ Paper Code 2024 linked, not harvested report
39 Orca-Math 7B (fine-tuned) 86.87 ✓ Paper – 2024 no code linked report
40 DART-Math-DSMath-7B-Prop2Diff (0-shot CoT, w/o code) 86.87 ✓ Paper Code 2024 official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 8 where Syntology's instrument failed) · 2 unverified report
41 OpenMath-CodeLlama-13B (w/ code, SC, k=50) 86.813 ✓ Paper Code 2024 linked, not harvested report
42 Gemini Pro (maj1@32) 86.5 – Paper Code 2023 linked, not harvested report
43 Codex (Self-Evaluation Guided Decoding, PAL, multiple reasoning chains, 9-shot gen, 5-shot eval) 85.5 – – – not matched report
44 Claude 1.3 (0-shot chain-of-thought) 85.2 – Paper – 2023 no code linked report
45 ToRA-Code-34B (SC, k=50) 85.134 ✓ Paper Code 2023 official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified report
46 OpenMath-CodeLlama-7B (w/ code, SC, k=50) 84.87 ✓ Paper Code 2024 linked, not harvested report
47 OVM-Mistral-7B (verify100@1) 84.77 – Paper Code 2023 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified report
48 OpenMath-Llama2-70B (w/ code) 84.770 ✓ Paper Code 2024 linked, not harvested report
49 OpenMath-CodeLlama-70B (w/ code) 84.670 ✓ Paper Code 2024 linked, not harvested report
50 code-davinci-002 175B (LEVER, 8-shot) 84.5175 – Paper Code 2023 official (archive's flag): 18 ran · 18 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 0 honoured, 2 violated, 12 with no contract checked; 4 where Syntology's instrument failed) · 4 unverified report
51 ToRA 70B 84.370 ✓ Paper Code 2023 official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified report
52 Shepherd + Mistral-7B (SFT on MetaMATH + PRM RL) 84.17 ✓ Paper Code 2023 linked, not harvested report
53 MathCoder-L-70B 83.970 ✓ Paper Code 2023 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified report
54 WizardMath-7B-V1.1 83.27 ✓ Paper Code 2023 10 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 3 where Syntology's instrument failed) · 6 unverified report
55 DIVERSE 175B (8-shot) 83.2175 – Paper – 2022 no code linked report
56 OVM-Mistral-7B (verify20@1) 82.67 – Paper Code 2023 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified report
57 DART-Math-Mistral-7B-Uniform (0-shot CoT, w/o code) 82.67 ✓ Paper Code 2024 official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 8 where Syntology's instrument failed) · 2 unverified report
58 ChatGPT (Ask, Refine, Trust) 82.6 – Paper – 2023 no code linked report
59 DART-Math-Llama3-8B-Uniform (0-shot CoT, w/o code) 82.58 ✓ Paper Code 2024 official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 8 where Syntology's instrument failed) · 2 unverified report
60 MetaMath 70B 82.370 ✓ Paper Code 2023 official (archive's flag): 15 ran · 15 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 14 where Syntology's instrument failed) · 7 unverified report
61 MuggleMATH 70B 82.370 ✓ Paper Code 2023 linked, not harvested report
62 PaLM 540B (Self Improvement, Self Consistency) 82.1540 – Paper – 2022 no code linked report
63 MathCoder-CL-34B 81.734 ✓ Paper Code 2023 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified report
64 WizardMath-70B-V1.0 81.670 ✓ Paper Code 2023 10 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 3 where Syntology's instrument failed) · 6 unverified report
65 Phi-GSM+V 1.3B+1.3B (verify48@1) 81.52.6 – Paper – 2023 no code linked report
66 DART-Math-Mistral-7B-Prop2Diff (0-shot CoT, w/o code) 81.17 ✓ Paper Code 2024 official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 8 where Syntology's instrument failed) · 2 unverified report
67 DART-Math-Llama3-8B-Prop2Diff (0-shot CoT, w/o code) 81.18 ✓ Paper Code 2024 official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 8 where Syntology's instrument failed) · 2 unverified report
68 Claude Instant 1.1 (0-shot chain-of-thought) 80.9 – Paper – 2023 no code linked report
69 ToRA-Code 34B 80.734 ✓ Paper Code 2023 official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified report
70 OpenMath-CodeLlama-34B (w/ code) 80.734 ✓ Paper Code 2024 linked, not harvested report
71 PaLM 2 (few-shot, k=8, CoT) 80.7 – Paper Code 2023 linked, not harvested report
72 MMOS-DeepSeekMath-7B(0-shot) 80.57 ✓ Paper Code 2024 official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified report
73 MMOS-CODE-34B(0-shot) 80.434 ✓ Paper Code 2024 official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified report
74 OpenMath-Mistral-7B (w/ code) 80.27 ✓ Paper Code 2024 linked, not harvested report
75 Self-Evaluation Guided Decoding (Codex, PAL, single reasoning chain, 9-shot gen, 5-shot eval) 80.2 – – – not matched report
76 OpenMath-CodeLlama-13B (w/ code) 78.813 ✓ Paper Code 2024 linked, not harvested report
77 Minerva 540B (CoT) 78.5540 – Paper Code 2022 linked, not harvested report
78 Camelidae-8×34B (5-shot) 78.3 – Paper Code 2024 linked, not harvested report
79 Qwen2idae-16x14B (5-shot) 77.8 – Paper Code 2024 linked, not harvested report
80 MetaMath-Mistral-7B 77.77 ✓ Paper Code 2023 official (archive's flag): 15 ran · 15 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 14 where Syntology's instrument failed) · 7 unverified report
81 OpenChat-3.5 7B 77.37 – Paper Code 2023 official: harvested, nothing ran · 0 ran · 1 unverified report
82 DeepMind 70B Model (STaR, maj1@96) 76.570 ✓ Paper – 2022 no code linked report
83 Arithmo2-Mistral-7B 76.47 – – – not matched report
84 OpenMath-CodeLlama-7B (w/ code) 75.97 ✓ Paper Code 2024 linked, not harvested report
85 ToRA-Code 13B 75.813 ✓ Paper Code 2023 official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified report
86 Arithmo-Mistral-7B 74.77 – – – not matched report
87 PaLM 540B maj1@40 (8-shot) 74.4540 ✓ Paper Code 2022 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result report
88 PaLM 540B (Self Consistency) 74.4540 – Paper – 2022 no code linked report
89 Phi-GSM 2.7B (fine-tuned) 74.32.7 – Paper – 2023 no code linked report
90 MathCoder-CL-13B 74.17 ✓ Paper Code 2023 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified report
91 MuggleMATH 13B 7413 ✓ Paper Code 2023 linked, not harvested report
92 MMOS-CODE-7B(0-shot) 73.97 ✓ Paper Code 2024 official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified report
93 CodeT5+ 73.80.77 – Paper Code 2023 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified report
94 Llama-3.3-70B + CAPO 73.73 – Paper Code 2025 linked, not harvested report
95 OVM-Llama2-7B (verify100@1) 73.77 – Paper Code 2023 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified report
96 PaLM 540B (Self Improvement, CoT Prompting) 73.5540 – Paper – 2022 no code linked report
97 KwaiYiiMath 13B 73.313 ✓ Paper – 2023 no code linked report
98 ToRA-Code 7B 72.67 ✓ Paper Code 2023 official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 6 where Syntology's instrument failed) · 1 unverified report
99 MathCoder-L-13B 72.613 ✓ Paper Code 2023 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified report
100 DBRX Base 132B 72.3 – – – not matched report
101 Self-Evaluation Guided Decoding (Codex, CoT, single reasoning chain, 9-shot gen, 5-shot eval) 71.9 – – – not matched report
102 MetaMath 13B 71.013 ✓ Paper Code 2023 official (archive's flag): 15 ran · 15 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 14 where Syntology's instrument failed) · 7 unverified report
103 MuggleMATH 7B 69.87 ✓ Paper Code 2023 linked, not harvested report
104 LLaMA 65B-maj1@k 69.765 – Paper Code 2023 official: no sample here; runs from other or unrecorded repositories · 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified report
105 Minerva 62B (maj1@100) 68.562 ✓ Paper Code 2022 linked, not harvested report
106 code-davinci-002 (Least-to-Most Prompting) 68.01175 – Paper Code 2022 linked, not harvested report
107 MathCoder-CL-7B 67.87 ✓ Paper Code 2023 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified report
108 DBRX Instruct 132B 66.9 – – – not matched report
109 MetaMath 7B 66.47 ✓ Paper Code 2023 official (archive's flag): 15 ran · 15 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 14 where Syntology's instrument failed) · 7 unverified report
110 Mistral-Small-24B + CAPO 65.07 – Paper Code 2025 linked, not harvested report
111 RFT 70B 64.879 ✓ Paper Code 2023 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified report
112 MathCoder-L-7B 64.27 ✓ Paper Code 2023 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified report
113 WizardMath-13B-V1.0 63.913 ✓ Paper Code 2023 10 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 3 where Syntology's instrument failed) · 6 unverified report
114 GPT-J (CoRe) 63.212 – Paper Code 2022 linked, not harvested report
115 Llama-2 70B (on 100 first questions, 4-shot, auto-optimized prompting) 6170 – Paper – 2024 no code linked report
116 Qwen2.5-32B + CAPO 60.2 – Paper Code 2025 linked, not harvested report
117 LLaMA 2 70B (CoT-Influx) 59.5970 – Paper – 2023 no code linked report
118 Orca 2 13B 59.1413 – Paper – 2023 no code linked report
119 U-PaLM 58.5540 – Paper – 2022 no code linked report
120 PaLM-540B (few-Shot-cot) 58.1540 ✓ Paper Code 2022 official: harvested, nothing ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified report
121 GPT-3.5 (few-shot, k=5) 57.1 – Paper Code 2023 community repositories only · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified report
122 Minerva 8B (maj5@100) 56.88 – Paper Code 2022 linked, not harvested report
123 LLaMA 2 70B (on-shot) 56.870 – Paper Code 2023 community repositories only · 33 ran (of which 7 constructed an object rather than computing a result; 20 with no instrument failure: 1 honoured, 1 violated, 18 with no contract checked; 13 where Syntology's instrument failed) · 19 unverified report
124 PaLM 540B (8-shot) 56.5540 ✓ Paper Code 2022 linked, not harvested report
125 PaLM 540B (CoT Prompting) 56.5540 – Paper – 2022 no code linked report
126 RFT 13B 55.313 ✓ Paper Code 2023 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified report
127 Finetuned GPT-3 175B + verifier 55.0175 ✓ Paper Code 2022 official: harvested, nothing ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified report
128 WizardMath-7B-V1.0 54.97 ✓ Paper Code 2023 10 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 3 where Syntology's instrument failed) · 6 unverified report
129 LLaMA 33B-maj1@k 53.133 – Paper Code 2023 official: no sample here; runs from other or unrecorded repositories · 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified report
130 Minerva 62B (8-shot) 52.462 ✓ Paper Code 2022 linked, not harvested report
131 Mistral 7B (maj@8) 52.27 – Paper Code 2023 official (archive's flag): 9 ran · 10 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified report
132 Llemma 34B 51.534 – Paper Code 2023 official (archive's flag): 6 ran · 7 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified report
133 Text-davinci-002-175B (zero-plus-few-Shot-cot (8 samples)) 51.5175 ✓ Paper Code 2022 official: harvested, nothing ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified report
134 RFT 7B 51.27 ✓ Paper Code 2023 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified report
135 LLaMA 65B 50.965 – Paper Code 2023 official: no sample here; runs from other or unrecorded repositories · 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified report
136 Orca 2 7B 47.237 – Paper – 2023 no code linked report
137 Llama-2 13B (on 100 first questions, 4-shot, auto-optimized prompting) 4313 – Paper – 2024 no code linked report
138 text-davinci-002 175B (2-shot, CoT) 41.3175 ✓ Paper Code 2022 official: harvested, nothing ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified report
139 Mistral 7B (on 100 first questions, 4-shot, auto-optimized prompting) 417 – Paper – 2024 no code linked report
140 text-davinci-002 175B (0-shot, CoT) 40.7175 ✓ Paper Code 2022 official: harvested, nothing ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified report
141 Branch-Train-MiX 4x7B (sampling top-2 experts) 37.1 – Paper Code 2024 linked, not harvested report
142 Llemma 7B 36.47 – Paper Code 2023 official (archive's flag): 6 ran · 7 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified report
143 LLaMA 33B 35.633 – Paper Code 2023 official: no sample here; runs from other or unrecorded repositories · 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified report
144 Vicuna (SYRELM) 35.213 ✓ Paper Code 2023 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified report
145 PaLM 62B (8-shot) 33.062 ✓ Paper Code 2022 linked, not harvested report
146 PaLM 540B (Self Improvement, Standard-Prompting) 32.2540 – Paper – 2022 no code linked report
147 LLaMA 13B-maj1@k 29.313 – Paper Code 2023 official: no sample here; runs from other or unrecorded repositories · 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified report
148 Minerva 8B-maj1@k (8-shot) 28.48 ✓ Paper Code 2022 linked, not harvested report
149 GPT-2-Medium 355M + question-solution classifier (BS=5) 20.80.355 – Paper – 2022 no code linked report
150 GPT-Neo-2.7B + Self-Sampling 19.52.7 – Paper Code 2022 official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified report
151 GPT-2-Medium 355M (fine-tuned, BS=5) 18.30.355 – Paper – 2022 no code linked report
152 LLaMA 7B (maj1@k) 18.17 – Paper Code 2023 official: no sample here; runs from other or unrecorded repositories · 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified report
153 PaLM 540B (few-shot) 17.9540 ✓ Paper Code 2022 official: harvested, nothing ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified report
154 PaLM 540B (Standard-Prompting) 17.9540 – Paper – 2022 no code linked report
155 LLaMA 13B 17.813 – Paper Code 2023 official: no sample here; runs from other or unrecorded repositories · 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified report
156 GPT-2-Medium 355M + question-solution classifier (BS=1) 16.80.355 – Paper – 2022 no code linked report
157 Minerva 8B (8-shot) 16.28 ✓ Paper Code 2022 linked, not harvested report
158 GPT-2-Medium 355M (BS=5) 12.20.355 – Paper – 2022 no code linked report
159 LLaMA 7B 11.07 – Paper Code 2023 official: no sample here; runs from other or unrecorded repositories · 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified report
160 Text-davinci-002-175B (0-shot) 10.4175 ✓ Paper Code 2022 official: harvested, nothing ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified report
161 GPT-Neo 125M + Self-Sampling 7.50.125 – Paper Code 2022 official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified report
162 UL2 20B (chain-of-thought) 4.420 – Paper Code 2022 community repositories only · 15 ran (of which 0 constructed an object rather than computing a result; 15 with no instrument failure: 0 honoured, 0 violated, 15 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified report
163 PaLM 8B (8-shot) 4.18 ✓ Paper Code 2022 linked, not harvested report
164 UL2 20B (0-shot) 4.120 – Paper Code 2022 community repositories only · 15 ran (of which 0 constructed an object rather than computing a result; 15 with no instrument failure: 0 honoured, 0 violated, 15 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified report

All 164 rows shown. 150 link to a paper page on this site; 85 are marked as using additional training data in the archive. No GitHub stars are tracked; "Code" is the first repository the archive lists for the row. The archive carries no row tags, review links or community-submitted rows for this table; none are shown. archive 2025-07-28

Syntology Ran reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”: of the code samples Syntology harvested from repositories linked to that row's paper (joined by arXiv id), N executed on a synthesized input and U are unverified (harvested, no recorded run). Of the N, C constructed an object rather than computing a result (a part of N, never taken away from it), K ran with no instrument failure (honoured or violated the contract Syntology drafted, or ran with no contract checked) and I are failures of Syntology's instrument, not of the code (a wrong contract draft, or a contract check that could not run). It counts code from repositories linked to that row's paper, not this result: the row's number was not reproduced and nothing here is a correctness claim. The other cell texts mean no graph line for the row: "linked, not harvested" (the archive links code, Syntology has not harvested it), "no code linked" (no code link in the archive), "not matched" (the row's paper URL matched no paper on this site). 77 rows have a graph line, from 28 distinct papers. Rows whose paper has a sample that ran: 76 (67 with a run with no instrument failure, 9 where every run was a failure of Syntology's instrument). Papers: 27 with a sample that ran: 25 with a run with no instrument failure, 2 where every run was a failure of Syntology's instrument. Whose code ran. When the archive marks a repository official for the row's paper, the cell starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code): “official (archive's flag): N ran” counts the samples from it that ran, instrument failures included; “official: harvested, nothing ran”; when it has no sample for this paper and other samples ran, “community repositories only” if every one came from a repository the archive lists for the paper but neither marks official nor records as mentioned in it, or one Syntology harvested outside the archive's links, and “official: no sample here; runs from other or unrecorded repositories” if any came from a repository named in the paper or found in its text by Syntology, or has no repository recorded; “official: harvested for another paper” (no sample from it is linked to this paper) or “official: not harvested”. No state is printed when the archive marks none official. The cell's tooltip names the repositories the samples that ran came from. Of the 77 rows with a graph line, 67 have an archive-official repository: on 47 a sample from it ran (38 with a run with no instrument failure), on 4 the samples that ran came from community repositories only, and on 8 from other or unrecorded repositories. The filters above hide rows and never renumber them; the first two count, by default, only a run with no instrument failure, and offer any run, instrument failures included. Counting each paper once, of 370 samples: 277 ran (of which 18 constructed an object rather than computing a result; 163 with no instrument failure: 11 honoured, 14 violated, 138 with no contract checked; 114 where Syntology's instrument failed) · 93 unverified. Separately, 115 of those 370 are pointer-only (licence): the site points at that code rather than redistributing it, a licence property recorded for ran and unverified samples alike; each cell's tooltip carries the row's own pointer-only count. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. Per-sample status is on the paper page; each Ran cell links to that paper's sample list. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.

Since the archive Syntology

Of the 8,888 newer papers whose tables Syntology has read (8,886 checked in full and 2 with a leaderboard still awaiting a verdict), 230 had this board among the leaderboards the extractor considered for their tables (it keeps at most 10 boards per paper and 4 per dataset; the lists of 1,484 of the 8,888 were already full without this board, so if it was crowded out of them it is not counted), and 168 proposed a result; of those 168, 127 gave no cell that passed the extractor's own checks, 41 were rejected by the independent check, 0 await a verdict and 0 were placed. Syntology has checked 8,886 of the 18,637 papers on this site that are newer than the archive (for 3,717 of them no archive leaderboard matched the paper's tables, so there was nothing further to check); 733 were read and have nothing to place (arXiv has no HTML version of the paper, or that version has no tables), 41 could not be read (the extractor's reply could not be parsed), and results from the other 8,977 appear after they are checked.

Newer papers whose tables Claude Sonnet 4.5 judged to report on this benchmark (a model's judgement, not a result; newest first by arXiv id)

Paper arXiv date Ran Syntology Outcome
SciWalker: Synthesizing Scientific Coding Problems with Operator Graphs and Execution Feedback arXiv:2609.30054 24 Sep 2026 repository linked, no samples harvested rejected by the independent check
DRT: Dense Reasoning Trace for Efficient and Grounded Multimodal Reasoning arXiv:2609.21675 18 Sep 2026 repository linked, no samples harvested rejected by the independent check
Chain-of-Thought Entropy as a Reliability Signal: A Preregistered Reproduction arXiv:2609.19606 17 Sep 2026 repository linked, no samples harvested refused by a rule: the value is not on the scale of the leaderboard column's archive values
Register Tokens for Bounded-State Reasoning in Diffusion Language Models arXiv:2609.16372 14 Sep 2026 repository linked, no samples harvested refused by a rule: the model gave no value to check against the cell
ZGCM-1: A Fully Open and Extremely Efficient Foundation Model for Math and Agentic Search arXiv:2609.13356 11 Sep 2026 9 harvested: 9 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 4 honoured, 0 violated, 0 with no contract checked; 5 where Syntology's instrument failed) · 0 unverified refused by a rule: the table's column header does not name the leaderboard's metric
Chopthin-Consensus Power Sampling: A Diversity-Preserving Approach to LLM Decoding arXiv:2609.12243 10 Sep 2026 15 harvested: 14 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 2 honoured, 2 violated, 3 with no contract checked; 7 where Syntology's instrument failed) · 1 unverified rejected by the independent check
Codebook Agent: Amortized Topology Design for LLM Multi-Agent Systems arXiv:2609.02264 2 Sep 2026 18 harvested: 16 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 3 honoured, 2 violated, 2 with no contract checked; 9 where Syntology's instrument failed) · 2 unverified refused by a rule: the number the model quoted is not the number printed in the cell it pointed at
Fine-Tuning Low-Bit Models with Gradient in Quantized Code Space arXiv:2608.30908 31 Aug 2026 14 harvested: 14 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 4 honoured, 3 violated, 1 with no contract checked; 6 where Syntology's instrument failed) · 0 unverified refused by a rule: the number the model quoted is not the number printed in the cell it pointed at
SPEAR: Distilling Domain-Adaptive Reasoning Skeletons via Sequential Symbolic... arXiv:2608.26550 27 Aug 2026 6 harvested: 5 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 5 where Syntology's instrument failed) · 1 unverified refused by a rule: the table's column header does not name the leaderboard's metric
Low-Rank Ternary Adaptation for Fine-Tuning Transformers arXiv:2608.24469 25 Aug 2026 10 harvested: 10 ran (of which 3 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 0 violated, 3 with no contract checked; 5 where Syntology's instrument failed) · 0 unverified rejected by the independent check
Dual-Adversarial Safety Alignment: Cultivating Intrinsic Threat Comprehension in LRMs arXiv:2608.09542 10 Aug 2026 22 harvested: 20 ran (of which 20 constructed an object rather than computing a result; 20 with no instrument failure: 0 honoured, 0 violated, 20 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified; every one of the 20 samples that ran constructed an object rather than computing a result rejected by the independent check
Commit Locally, Exit Globally: Coordinating Adaptive Sampling and Early Exit in... arXiv:2607.28166 30 Jul 2026 15 harvested: 15 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 1 honoured, 3 violated, 4 with no contract checked; 7 where Syntology's instrument failed) · 0 unverified rejected by the independent check
Reasoning Fine-Tuning Induces Persistent Latent Policy States arXiv:2607.18532 20 Jul 2026 1 harvested: 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified refused by rules: the model pointed at a header row; the model gave no value to check against the cell
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling arXiv:2607.16252 27 Jun 2026 4 harvested: 4 ran (of which 3 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified rejected by the independent check
Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers arXiv:2606.31779 30 Jun 2026 repository linked, no samples harvested rejected by the independent check
Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs arXiv:2606.31413 30 Jun 2026 3 harvested: 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 2 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified refused by a rule: the table's column header does not name the leaderboard's metric
Dynamic-dLLM: Dynamic Cache-Budget and Adaptive Parallel Decoding for Training-Free... arXiv:2606.26120 27 May 2026 10 harvested: 8 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 1 violated, 0 with no contract checked; 6 where Syntology's instrument failed) · 2 unverified refused by a rule: the row is an ablation variant, not the method itself
PEAR: Permutation-Equivariant Adaptive Routing Multi-Agent Debate arXiv:2606.20621 26 May 2026 7 harvested: 6 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 3 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified refused by a rule: the value is not on the scale of the leaderboard column's archive values
Muonᵖ: Muon with Fractional Spectral Powers arXiv:2606.13867 11 Jun 2026 held, no recorded run refused by a rule: the table's column header does not name the leaderboard's metric
Fine-tuning Multi-modal LLMs with ART: Art-based Reinforcement Training arXiv:2606.11854 10 Jun 2026 15 harvested: 11 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 11 where Syntology's instrument failed) · 4 unverified refused by a rule: the model gave no value to check against the cell
SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs arXiv:2606.09886 3 Jun 2026 2 harvested: 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified refused by a rule: one configuration gave two values for one column
Artificial Intelligence for Mathematical Reasoning: An Integrated Survey of Language... arXiv:2606.08728 7 Jun 2026 repository linked, no samples harvested refused by rules: the model pointed at a header row; the row label the model quoted is not the row's label; the table, row or column pointed at does not exist
AsyncLane: Decoupling Refinement from Advancement in Diffusion Language Model Decoding arXiv:2606.08411 7 Jun 2026 repository linked, no samples harvested refused by a rule: one configuration gave two values for one column
TARPO: Token-Wise Latent-Explicit Reasoning via Action-Routing Policy Optimization arXiv:2606.05859 4 Jun 2026 18 harvested: 10 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 1 violated, 1 with no contract checked; 8 where Syntology's instrument failed) · 8 unverified refused by rules: the table's column header does not name the leaderboard's metric; one configuration gave two values for one column
SemBlock: Semantic Boundary Dynamic Blocks for Diffusion LLMs arXiv:2606.04964 3 Jun 2026 2 harvested: 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified refused by a rule: the table's column header does not name the leaderboard's metric
From Layers to Submodules: Rethinking Granularity in Replacement-Based LLM Compression arXiv:2606.02559 1 Jun 2026 30 harvested: 14 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 4 honoured, 0 violated, 1 with no contract checked; 9 where Syntology's instrument failed) · 16 unverified refused by a rule: one configuration gave two values for one column
MOC: Multi-Order Communication in LLM-based Multi-Agent Systems arXiv:2606.02359 1 Jun 2026 8 harvested: 8 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 1 honoured, 0 violated, 5 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified refused by rules: the table's column header does not name the leaderboard's metric; the quoted value appears more than once in the row
BlockGen: Flexible Blockwise Sequence Modeling with Hybrid Samplers arXiv:2606.02241 1 Jun 2026 18 harvested: 12 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 2 violated, 1 with no contract checked; 8 where Syntology's instrument failed) · 6 unverified refused by a rule: one configuration gave two values for one column
ProjQ: Project-and-Quantize for Adapter-Aware LLM Compression arXiv:2606.00494 30 May 2026 6 harvested: 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 3 unverified refused by a rule: the row label the model quoted is not the row's label
D³: Dynamic Directional Graph-Constrained Data Scheduling for LLM Training arXiv:2605.31164 29 May 2026 repository linked, no samples harvested refused by a rule: the row label the model quoted is not the row's label
NaRA: Noise-Aware LoRA for Parameter-Efficient Fine-Tuning of Diffusion LLMs arXiv:2605.29716 28 May 2026 5 harvested: 4 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 1 unverified refused by a rule: one configuration gave two values for one column
TRACE: Toulmin-based Reasoning Assessment through Constructive Elements for LLM CoT Evaluation arXiv:2605.29656 28 May 2026 4 harvested: 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 3 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified rejected by the independent check
GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models arXiv:2605.29398 28 May 2026 13 harvested: 11 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 2 honoured, 1 violated, 1 with no contract checked; 7 where Syntology's instrument failed) · 2 unverified refused by a rule: the row is an ablation variant, not the method itself
ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inference arXiv:2605.27081 26 May 2026 repository linked, no samples harvested refused by a rule: the table's column header does not name the leaderboard's metric
Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped... arXiv:2605.26733 26 May 2026 3 harvested: 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified refused by rules: the table's column header does not name the leaderboard's metric; one configuration gave two values for one column
Selective Latent Thinking: Adaptive Compression of LLM Reasoning Chains arXiv:2605.25745 25 May 2026 repository linked, no samples harvested refused by rules: the table does not name the leaderboard's dataset; the table's column header does not name the leaderboard's metric
Hide to Guide: Learning via Semantic Masking arXiv:2605.25198 24 May 2026 10 harvested: 5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 1 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 5 unverified refused by a rule: the table's column header does not name the leaderboard's metric
Summoning the Oracle to Slay It: Mitigating Look-Ahead Bias in Financial Backtesting... arXiv:2605.24564 23 May 2026 6 harvested: 5 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 5 where Syntology's instrument failed) · 1 unverified refused by a rule: the model gave no value to check against the cell
Adaptive Mass-Segmented KV Compression for Long-Context Reasoning arXiv:2605.23200 22 May 2026 7 harvested: 5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified refused by a rule: the row label the model quoted is not the row's label
Learnability-Informed Fine-Tuning of Diffusion Language Models arXiv:2605.22939 21 May 2026 held, no recorded run rejected by the independent check
FuRA: Full-Rank Parameter-Efficient Fine-Tuning with Spectral Preconditioning arXiv:2605.22869 19 May 2026 6 harvested: 6 ran (of which 1 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 1 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified refused by a rule: the table's column header does not name the leaderboard's metric
TextReg: Mitigating Prompt Distributional Overfitting via Regularized Text-Space Optimization arXiv:2605.21318 20 May 2026 4 harvested: 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified refused by a rule: the number the model quoted is not the number printed in the cell it pointed at
AGPO: Adaptive Group Policy Optimization with Dual Statistical Feedback arXiv:2605.20722 20 May 2026 5 harvested: 4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified rejected by the independent check
CopT: Contrastive On-Policy Thinking with Continuous Spaces for General and Agentic Reasoning arXiv:2605.20075 19 May 2026 11 harvested: 6 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 6 where Syntology's instrument failed) · 5 unverified refused by a rule: one configuration gave two values for one column
Scientific Logicality Enriched Methodology for LLM Reasoning: A Practice in Physics arXiv:2605.17104 16 May 2026 7 harvested: 6 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 5 where Syntology's instrument failed) · 1 unverified refused by a rule: the table's column header does not name the leaderboard's metric
Learning-Zone Energy: Online Data Selection for Efficient RL Post-Training arXiv:2605.17003 16 May 2026 held, no recorded run refused by rules: the value is not on the scale of the leaderboard column's archive values; the number the model quoted is not the number printed in the cell it pointed at
Roll Out and Roll Back: Diffusion LLMs are Their Own Efficiency Teachers arXiv:2605.16941 16 May 2026 13 harvested: 12 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 0 violated, 1 with no contract checked; 9 where Syntology's instrument failed) · 1 unverified rejected by the independent check
MixSD: Mixed Contextual Self-Distillation for Knowledge Injection arXiv:2605.16865 16 May 2026 repository linked, no samples harvested refused by a rule: the table's column header does not name the leaderboard's metric
FlowCompile: An Optimizing Compiler for Structured LLM Workflows arXiv:2605.13647 13 May 2026 13 harvested: 10 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 7 where Syntology's instrument failed) · 3 unverified rejected by the independent check
CANTANTE: Optimizing Agentic Systems via Contrastive Credit Attribution arXiv:2605.13295 13 May 2026 7 harvested: 6 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified refused by a rule: the number the model quoted is not the number printed in the cell it pointed at
Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion arXiv:2605.12825 12 May 2026 11 harvested: 10 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 5 where Syntology's instrument failed) · 1 unverified refused by a rule: the table's column header does not name the leaderboard's metric
Towards Order Fairness: Mitigating LLMs Order Sensitivity through Dual Group Advantage... arXiv:2605.11974 12 May 2026 15 harvested: 10 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 1 violated, 1 with no contract checked; 8 where Syntology's instrument failed) · 5 unverified refused by a rule: the table does not name the leaderboard's dataset
Block-R1: Rethinking the Role of Block Size in Multi-domain Reinforcement Learning for... arXiv:2605.11726 12 May 2026 5 harvested: 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified refused by rules: the table's column header does not name the leaderboard's metric; the row is an ablation variant, not the method itself
Toward Stable Value Alignment: Introducing Independent Modules for Consistent Value Guidance arXiv:2605.11712 12 May 2026 1 harvested: 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result rejected by the independent check
Language Modeling with Hyperspherical Flows arXiv:2605.11125 11 May 2026 24 harvested: 21 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 3 honoured, 1 violated, 2 with no contract checked; 15 where Syntology's instrument failed) · 3 unverified rejected by the independent check
Breaking the Reward Barrier: Accelerating Tree-of-Thought Reasoning via Speculative Exploration arXiv:2605.10195 11 May 2026 15 harvested: 14 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 2 violated, 0 with no contract checked; 11 where Syntology's instrument failed) · 1 unverified refused by a rule: the number the model quoted is not the number printed in the cell it pointed at
Team-Based Self-Play With Dual Adaptive Weighting for Fine-Tuning LLMs arXiv:2605.09922 11 May 2026 repository linked, no samples harvested refused by a rule: the row is an ablation variant, not the method itself
RADAR: Redundancy-Aware Diffusion for Multi-Agent Communication Structure Generation arXiv:2605.09907 11 May 2026 10 harvested: 5 ran (of which 1 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 1 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 5 unverified refused by a rule: the number the model quoted is not the number printed in the cell it pointed at
TAD: Temporal-Aware Trajectory Self-Distillation for Fast and Accurate Diffusion LLM arXiv:2605.09536 10 May 2026 9 harvested: 8 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 2 honoured, 0 violated, 2 with no contract checked; 4 where Syntology's instrument failed) · 1 unverified refused by a rule: one configuration gave two values for one column
RuPLaR : Efficient Latent Compression of LLM Reasoning Chains with Rule-Based Priors... arXiv:2605.09346 10 May 2026 3 harvested: 2 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified rejected by the independent check
DARE: Difficulty-Adaptive Reinforcement Learning with Co-Evolved Difficulty Estimation arXiv:2605.09188 9 May 2026 held, no recorded run refused by a rule: the number the model quoted is not the number printed in the cell it pointed at
DARE: Diffusion Language Model Activation Reuse for Efficient Inference arXiv:2605.08134 1 May 2026 repository linked, no samples harvested refused by a rule: the table's column header does not name the leaderboard's metric
LaTER: Efficient Test-Time Reasoning via Latent Exploration and Explicit Verification arXiv:2605.07315 8 May 2026 20 harvested: 18 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 1 violated, 2 with no contract checked; 14 where Syntology's instrument failed) · 2 unverified rejected by the independent check
Safety Anchor: Defending Harmful Fine-tuning via Geometric Bottlenecks arXiv:2605.05995 7 May 2026 repository linked, no samples harvested refused by a rule: the table's column header does not name the leaderboard's metric
Correct Is Not Enough: Training Reasoning Planners with Executor-Grounded Rewards arXiv:2605.03862 5 May 2026 3 harvested: 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified rejected by the independent check
Break the Block: Dynamic-size Reasoning Blocks for Diffusion Large Language Models via... arXiv:2605.02263 4 May 2026 10 harvested: 9 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 2 honoured, 4 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified refused by a rule: the row is an ablation variant, not the method itself
Stochastic Sparse Attention for Memory-Bound Inference arXiv:2605.01910 3 May 2026 3 harvested: 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified; the one sample that ran constructed an object rather than computing a result refused by a rule: the number the model quoted is not the number printed in the cell it pointed at
Diversity in Large Language Models under Supervised Fine-Tuning arXiv:2605.00195 30 Apr 2026 2 harvested: 2 ran (of which 2 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; every one of the 2 samples that ran constructed an object rather than computing a result refused by a rule: the table's column header does not name the leaderboard's metric
Dual-Track CoT: Budget-Aware Stepwise Guidance for Small LMs arXiv:2604.25039 27 Apr 2026 3 harvested: 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified rejected by the independent check
DPRM: A Plug-in Doob h transform-induced Token-Ordering Module for Discrete Diffusion Models arXiv:2604.24357 27 Apr 2026 1 harvested: 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified refused by a rule: the table does not name the leaderboard's dataset
R²-dLLM: Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction arXiv:2604.18995 21 Apr 2026 13 harvested: 12 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 1 violated, 1 with no contract checked; 10 where Syntology's instrument failed) · 1 unverified refused by a rule: the row label the model quoted is not the row's label
DepCap: Adaptive Block-Wise Parallel Decoding for Efficient Diffusion LM Inference arXiv:2604.15750 17 Apr 2026 13 harvested: 12 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 1 violated, 1 with no contract checked; 9 where Syntology's instrument failed) · 1 unverified rejected by the independent check
Faster LLM Inference via Sequential Monte Carlo arXiv:2604.15672 17 Apr 2026 repository linked, no samples harvested rejected by the independent check
Adaptive Test-Time Compute Allocation for Reasoning LLMs via Constrained Policy Optimization arXiv:2604.14853 16 Apr 2026 repository linked, no samples harvested refused by a rule: the model gave no value to check against the cell
CausalDetox: Causal Head Selection and Intervention for Language Model Detoxification arXiv:2604.14602 16 Apr 2026 9 harvested: 6 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 3 unverified refused by rules: the value is not on the scale of the leaderboard column's archive values; the table's column header does not name the leaderboard's metric
Awakening Dormant Experts:Counterfactual Routing to Mitigate MoE Hallucinations arXiv:2604.14246 15 Apr 2026 repository linked, no samples harvested refused by a rule: the table's column header does not name the leaderboard's metric
Beyond Static Personas: Situational Personality Steering for Large Language Models arXiv:2604.13846 15 Apr 2026 7 harvested: 6 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 5 where Syntology's instrument failed) · 1 unverified refused by a rule: the table's column header does not name the leaderboard's metric
Training-Free Test-Time Contrastive Learning for Large Language Models arXiv:2604.13552 15 Apr 2026 14 harvested: 12 ran (of which 5 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 1 violated, 5 with no contract checked; 6 where Syntology's instrument failed) · 2 unverified rejected by the independent check
MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Models arXiv:2604.12928 14 Apr 2026 6 harvested: 6 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified refused by rules: the number the model quoted is not the number printed in the cell it pointed at; the row label the model quoted is not the row's label
Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model... arXiv:2604.12374 14 Apr 2026 24 harvested: 20 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 1 honoured, 3 violated, 4 with no contract checked; 12 where Syntology's instrument failed) · 4 unverified refused by a rule: the table's column header does not name the leaderboard's metric
Pioneer Agent: Continual Improvement of Small Language Models in Production arXiv:2604.09791 10 Apr 2026 repository linked, no samples harvested refused by a rule: the row label the model quoted is not the row's label
DMax: Aggressive Parallel Decoding for dLLMs arXiv:2604.08302 9 Apr 2026 3 harvested: 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified rejected by the independent check
Expert-Choice Routing Enables Adaptive Computation in Diffusion Language Models arXiv:2604.01622 2 Apr 2026 5 harvested: 5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 1 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified rejected by the independent check
Finding and Reactivating Post-Trained LLMs' Hidden Safety Mechanisms arXiv:2604.00012 10 Mar 2026 1 harvested: 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified rejected by the independent check
AMALIA Technical Report: A Fully Open Source Large Language Model for European Portuguese arXiv:2603.26511 27 Mar 2026 repository linked, no samples harvested refused by rules: the model gave no value to check against the cell; the table's column header does not name the leaderboard's metric
S2D2: Fast Decoding for Diffusion LLMs via Training-Free Self-Speculation arXiv:2603.25702 26 Mar 2026 1 harvested: 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified refused by a rule: one configuration gave two values for one column
WIST: Web-Grounded Iterative Self-Play Tree for Domain-Targeted Reasoning Improvement arXiv:2603.22352 22 Mar 2026 19 harvested: 13 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 2 honoured, 0 violated, 2 with no contract checked; 9 where Syntology's instrument failed) · 6 unverified refused by a rule: the row is an ablation variant, not the method itself
Scaling Attention via Feature Sparsity arXiv:2603.22300 17 Mar 2026 repository linked, no samples harvested refused by rules: the table does not name the leaderboard's dataset; the number the model quoted is not the number printed in the cell it pointed at
ROM: Real-time Overthinking Mitigation via Streaming Detection and Intervention arXiv:2603.22016 23 Mar 2026 5 harvested: 3 ran (of which 1 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 1 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified refused by a rule: the row label the model quoted is not the row's label
Hear Both Sides: Efficient Multi-Agent Debate via Diversity-Aware Message Retention arXiv:2603.20640 21 Mar 2026 11 harvested: 8 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 5 where Syntology's instrument failed) · 3 unverified refused by a rule: the model gave no value to check against the cell
Multi-Agent Debate with Memory Masking arXiv:2603.20215 3 Mar 2026 6 harvested: 5 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 1 unverified refused by a rule: the number the model quoted is not the number printed in the cell it pointed at
Decoding the Critique Mechanism in Large Reasoning Models arXiv:2603.16331 17 Mar 2026 1 harvested: 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified rejected by the independent check
IGU-LoRA: Adaptive Rank Allocation via Integrated Gradients and Uncertainty-Aware Scoring arXiv:2603.13792 14 Mar 2026 held, no recorded run refused by a rule: one configuration gave two values for one column
GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuning arXiv:2603.10243 10 Mar 2026 10 harvested: 9 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 3 honoured, 0 violated, 1 with no contract checked; 5 where Syntology's instrument failed) · 1 unverified refused by a rule: one configuration gave two values for one column
ES-dLLM: Efficient Inference for Diffusion Large Language Models by Early-Skipping arXiv:2603.10088 10 Mar 2026 2 harvested: 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified rejected by the independent check
SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization arXiv:2603.08185 9 Mar 2026 6 harvested: 6 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 6 where Syntology's instrument failed) · 0 unverified refused by a rule: the table's column header does not name the leaderboard's metric
DyLLM: Efficient Diffusion LLM Inference via Saliency-based Token Selection and... arXiv:2603.08026 9 Mar 2026 1 harvested: 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified refused by a rule: one configuration gave two values for one column
Exact Is Easier: Credit Assignment for Cooperative LLM Agents arXiv:2603.06859 6 Mar 2026 10 harvested: 8 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 8 where Syntology's instrument failed) · 2 unverified refused by a rule: the row is an ablation variant, not the method itself
Post-Training Language Models for Crosslingual Consistency arXiv:2603.04678 4 Mar 2026 repository linked, no samples harvested refused by a rule: the row is an ablation variant, not the method itself
MASPOB: Bandit-Based Prompt Optimization for Multi-Agent Systems with Graph Neural Networks arXiv:2603.02630 3 Mar 2026 repository linked, no samples harvested rejected by the independent check

Syntology 168 newer papers judged to report on this board: the papers counted above as having proposed a result (0 placed); the first 100 are listed here, and 68 more in the JSON twin. The order is the arXiv identifier's, newest first, not a rank. The judgement is the model's, read from the paper's own tables; the outcome is what Syntology's checks did with it, and no value from a paper that was not placed is printed. “refused by a rule” names the extractor's rule; “no cell passed the extractor's checks” means the extractor's record holds no cell for this board, passed or refused. The 127 counted above as giving no cell that passed the extractor's own checks are the 127 marked “refused by a rule”. Drawn from 8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read).

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections