{"url":"/sota/math-word-problem-solving-on-math","task":{"name":"Math Word Problem Solving","url":"/task/math-word-problem-solving","note":null},"dataset":{"name":"MATH","url":"/dataset/math"},"category":"Reasoning","categories":["Reasoning"],"category_note":null,"description":"A math word problem is a mathematical exercise (such as in a textbook, worksheet, or exam) where significant background information on the problem is presented in ordinary language rather than in mathematical notation. As most word problems involve a narrative of some sort, they are sometimes referred to as story problems and may vary in the amount of technical language used.","description_from":"task","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","rank":"the archive's row order at snapshot; not re-ranked","rows_end_at":"2025-07-28","rows_withheld_as_spam":0,"metric_values":"the archive's strings, untouched"},"metrics":["Accuracy","Parameters (Billions)"],"metric_direction":{"note":"inferred from the metric name only (the archive records no direction); null = not inferred, chart draws points only","by_metric":{"Accuracy":"higher","Parameters (Billions)":null}},"counts":{"rows":135,"rows_with_code":121,"rows_with_paper_page":132,"rows_dated":132,"rows_using_additional_data":70},"rows":[{"rank_in_archive_order":1,"model":"Gemini 2.0 Flash Experimental","metrics":{"Accuracy":"89.7"},"uses_additional_data":false,"paper_date":null,"paper":null,"paper_url":null,"paper_title":"","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":2,"model":"Qwen2.5-Math-72B-Instruct(TIR,Greedy)","metrics":{"Accuracy":"88.1","Parameters (Billions)":"72"},"uses_additional_data":true,"paper_date":"2024-09-18","paper":"/paper/qwen2-5-math-technical-report-toward","paper_url":"https://arxiv.org/abs/2409.12122v1","paper_title":"Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":3,"model":"GPT-4 Turbo (MACM, w/code, voting)","metrics":{"Accuracy":"87.920"},"uses_additional_data":false,"paper_date":"2024-04-06","paper":"/paper/macm-utilizing-a-multi-agent-system-for","paper_url":"https://arxiv.org/abs/2404.04735v2","paper_title":"MACM: Utilizing a Multi-Agent System for Condition Mining in Solving Complex Mathematical Problems","code":"https://github.com/bin123apple/macm","n_code_links":1,"syntology":{"n_ran":0,"n_unverified":1,"n_samples":1,"n_pointer_only_licence":1}},{"rank_in_archive_order":4,"model":"Qwen2.5-Math-72B-Instruct(COT,Greedy)","metrics":{"Accuracy":"85.9","Parameters (Billions)":"72"},"uses_additional_data":true,"paper_date":"2024-09-18","paper":"/paper/qwen2-5-math-technical-report-toward","paper_url":"https://arxiv.org/abs/2409.12122v1","paper_title":"Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":5,"model":"Qwen2.5-Math-7B-Instruct(TIR,Greedy)","metrics":{"Accuracy":"85.2","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":"2024-09-18","paper":"/paper/qwen2-5-math-technical-report-toward","paper_url":"https://arxiv.org/abs/2409.12122v1","paper_title":"Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":6,"model":"GPT-4-code model (CSV, w/ code, SC, k=16)","metrics":{"Accuracy":"84.3"},"uses_additional_data":false,"paper_date":"2023-08-15","paper":"/paper/solving-challenging-math-word-problems-using","paper_url":"https://arxiv.org/abs/2308.07921v1","paper_title":"Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification","code":"https://github.com/kipok/nemo-skills","n_code_links":1,"syntology":null},{"rank_in_archive_order":7,"model":"Qwen2-Math-72B-Instruct(greedy)","metrics":{"Accuracy":"84.0","Parameters (Billions)":"72"},"uses_additional_data":true,"paper_date":"2024-07-15","paper":"/paper/qwen2-technical-report","paper_url":"https://arxiv.org/abs/2407.10671v4","paper_title":"Qwen2 Technical Report","code":"https://github.com/qwenlm/qwen1.5","n_code_links":6,"syntology":null},{"rank_in_archive_order":8,"model":"Qwen2.5-Math-7B-Instruct(COT,Greedy)","metrics":{"Accuracy":"83.6","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":"2024-09-18","paper":"/paper/qwen2-5-math-technical-report-toward","paper_url":"https://arxiv.org/abs/2409.12122v1","paper_title":"Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":9,"model":"Qwen2.5-Math-1.5B-Instruct(TIR,Greedy)","metrics":{"Accuracy":"79.9","Parameters (Billions)":"1.5"},"uses_additional_data":true,"paper_date":"2024-09-18","paper":"/paper/qwen2-5-math-technical-report-toward","paper_url":"https://arxiv.org/abs/2409.12122v1","paper_title":"Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":10,"model":"OpenMath2-Llama3.1-70B (majority@256)","metrics":{"Accuracy":"79.6"},"uses_additional_data":true,"paper_date":"2024-10-02","paper":"/paper/openmathinstruct-2-accelerating-ai-for-math","paper_url":"https://arxiv.org/abs/2410.01560v2","paper_title":"OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data","code":"https://github.com/NVIDIA/NeMo-Skills","n_code_links":1,"syntology":{"n_ran":0,"n_unverified":15,"n_samples":15,"n_pointer_only_licence":0}},{"rank_in_archive_order":11,"model":"OpenMath2-Llama3.1-8B (majority@256)","metrics":{"Accuracy":"76.1"},"uses_additional_data":true,"paper_date":"2024-10-02","paper":"/paper/openmathinstruct-2-accelerating-ai-for-math","paper_url":"https://arxiv.org/abs/2410.01560v2","paper_title":"OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data","code":"https://github.com/NVIDIA/NeMo-Skills","n_code_links":1,"syntology":{"n_ran":0,"n_unverified":15,"n_samples":15,"n_pointer_only_licence":0}},{"rank_in_archive_order":12,"model":"Qwen2.5-Math-1.5B-Instruct(COT,Greedy)","metrics":{"Accuracy":"75.8","Parameters (Billions)":"1.5"},"uses_additional_data":true,"paper_date":"2024-09-18","paper":"/paper/qwen2-5-math-technical-report-toward","paper_url":"https://arxiv.org/abs/2409.12122v1","paper_title":"Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":13,"model":"GPT-4-code model (CSV, w/ code)","metrics":{"Accuracy":"73.5"},"uses_additional_data":false,"paper_date":"2023-08-15","paper":"/paper/solving-challenging-math-word-problems-using","paper_url":"https://arxiv.org/abs/2308.07921v1","paper_title":"Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification","code":"https://github.com/kipok/nemo-skills","n_code_links":1,"syntology":null},{"rank_in_archive_order":14,"model":"CR (GPT-4-turbo model, w/ code)","metrics":{"Accuracy":"72.2"},"uses_additional_data":false,"paper_date":"2023-08-08","paper":"/paper/cumulative-reasoning-with-large-language","paper_url":"https://arxiv.org/abs/2308.04371v6","paper_title":"Cumulative Reasoning with Large Language Models","code":"https://github.com/iiis-ai/cumulative-reasoning","n_code_links":1,"syntology":{"n_ran":5,"n_unverified":1,"n_samples":6,"n_pointer_only_licence":6}},{"rank_in_archive_order":15,"model":"OpenMath2-Llama3.1-70B","metrics":{"Accuracy":"71.9"},"uses_additional_data":true,"paper_date":"2024-10-02","paper":"/paper/openmathinstruct-2-accelerating-ai-for-math","paper_url":"https://arxiv.org/abs/2410.01560v2","paper_title":"OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data","code":"https://github.com/NVIDIA/NeMo-Skills","n_code_links":1,"syntology":{"n_ran":0,"n_unverified":15,"n_samples":15,"n_pointer_only_licence":0}},{"rank_in_archive_order":16,"model":"LogicNet (with code interpreter)","metrics":{"Accuracy":"71.2"},"uses_additional_data":true,"paper_date":"2023-08-15","paper":"/paper/solving-challenging-math-word-problems-using","paper_url":"https://arxiv.org/abs/2308.07921v1","paper_title":"Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification","code":"https://github.com/kipok/nemo-skills","n_code_links":1,"syntology":null},{"rank_in_archive_order":17,"model":"Qwen2-72B-Instruct-Step-DPO (0-shot CoT, w/o code)","metrics":{"Accuracy":"70.8"},"uses_additional_data":true,"paper_date":"2024-06-26","paper":"/paper/step-dpo-step-wise-preference-optimization","paper_url":"https://arxiv.org/abs/2406.18629v1","paper_title":"Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs","code":"https://github.com/dvlab-research/step-dpo","n_code_links":1,"syntology":{"n_ran":7,"n_unverified":5,"n_samples":12,"n_pointer_only_licence":12}},{"rank_in_archive_order":18,"model":"GPT-4-code model (w/ code)","metrics":{"Accuracy":"69.7"},"uses_additional_data":false,"paper_date":"2023-08-15","paper":"/paper/solving-challenging-math-word-problems-using","paper_url":"https://arxiv.org/abs/2308.07921v1","paper_title":"Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification","code":"https://github.com/kipok/nemo-skills","n_code_links":1,"syntology":null},{"rank_in_archive_order":19,"model":"OpenMath2-Llama3.1-8B","metrics":{"Accuracy":"67.8"},"uses_additional_data":true,"paper_date":"2024-10-02","paper":"/paper/openmathinstruct-2-accelerating-ai-for-math","paper_url":"https://arxiv.org/abs/2410.01560v2","paper_title":"OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data","code":"https://github.com/NVIDIA/NeMo-Skills","n_code_links":1,"syntology":{"n_ran":0,"n_unverified":15,"n_samples":15,"n_pointer_only_licence":0}},{"rank_in_archive_order":20,"model":"AlphaMath-7B-SBS@3","metrics":{"Accuracy":"66.3"},"uses_additional_data":false,"paper_date":"2024-05-06","paper":"/paper/alphamath-almost-zero-process-supervision","paper_url":"https://arxiv.org/abs/2405.03553v3","paper_title":"AlphaMath Almost Zero: Process Supervision without Process","code":"https://github.com/MARIO-Math-Reasoning/Super_MARIO","n_code_links":1,"syntology":null},{"rank_in_archive_order":21,"model":"Minerva 62B (maj5@256)","metrics":{"Accuracy":"64.9","Parameters (Billions)":"62"},"uses_additional_data":false,"paper_date":"2022-06-29","paper":"/paper/solving-quantitative-reasoning-problems-with","paper_url":"https://arxiv.org/abs/2206.14858v2","paper_title":"Solving Quantitative Reasoning Problems with Language Models","code":"https://github.com/gair-nlp/abel","n_code_links":1,"syntology":null},{"rank_in_archive_order":22,"model":"DAMOMath-7B","metrics":{"Accuracy":"64.5","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":null,"paper":null,"paper_url":null,"paper_title":"","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":23,"model":"MMOS-DeepSeekMath-7B(0-shot,k=50)","metrics":{"Accuracy":"63.7","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":"2024-02-23","paper":"/paper/an-empirical-study-of-data-ability-boundary","paper_url":"https://arxiv.org/abs/2403.00799v1","paper_title":"An Empirical Study of Data Ability Boundary in LLMs' Math Reasoning","code":"https://github.com/cyzhh/MMOS","n_code_links":1,"syntology":{"n_ran":9,"n_unverified":2,"n_samples":11,"n_pointer_only_licence":11}},{"rank_in_archive_order":24,"model":"GPT-4-code model (w/o code)","metrics":{"Accuracy":"60.8"},"uses_additional_data":false,"paper_date":"2023-08-15","paper":"/paper/solving-challenging-math-word-problems-using","paper_url":"https://arxiv.org/abs/2308.07921v1","paper_title":"Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification","code":"https://github.com/kipok/nemo-skills","n_code_links":1,"syntology":null},{"rank_in_archive_order":25,"model":"OpenMath-CodeLlama-70B (w/ code, SC, k=50)","metrics":{"Accuracy":"60.4","Parameters (Billions)":"70"},"uses_additional_data":true,"paper_date":"2024-02-15","paper":"/paper/openmathinstruct-1-a-1-8-million-math","paper_url":"https://arxiv.org/abs/2402.10176v2","paper_title":"OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset","code":"https://github.com/kipok/nemo-skills","n_code_links":1,"syntology":null},{"rank_in_archive_order":26,"model":"OpenMath-CodeLlama-34B (w/ code, SC, k=50)","metrics":{"Accuracy":"60.2","Parameters (Billions)":"34"},"uses_additional_data":true,"paper_date":"2024-02-15","paper":"/paper/openmathinstruct-1-a-1-8-million-math","paper_url":"https://arxiv.org/abs/2402.10176v2","paper_title":"OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset","code":"https://github.com/kipok/nemo-skills","n_code_links":1,"syntology":null},{"rank_in_archive_order":27,"model":"ToRA-Code 34B model (w/ code, SC, k=50)","metrics":{"Accuracy":"60.0","Parameters (Billions)":"34"},"uses_additional_data":true,"paper_date":"2023-09-29","paper":"/paper/tora-a-tool-integrated-reasoning-agent-for","paper_url":"https://arxiv.org/abs/2309.17452v4","paper_title":"ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving","code":"https://github.com/microsoft/tora","n_code_links":1,"syntology":{"n_ran":5,"n_unverified":7,"n_samples":12,"n_pointer_only_licence":0}},{"rank_in_archive_order":28,"model":"DeepSeekMATH-RL-7B (w/ code, greedy decoding)","metrics":{"Accuracy":"58.8","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":"2024-02-05","paper":"/paper/deepseekmath-pushing-the-limits-of","paper_url":"https://arxiv.org/abs/2402.03300v3","paper_title":"DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models","code":"https://github.com/shibing624/medicalgpt","n_code_links":5,"syntology":{"n_ran":8,"n_unverified":16,"n_samples":24,"n_pointer_only_licence":0}},{"rank_in_archive_order":29,"model":"OpenMath-Llama2-70B (w/ code, SC, k=50)","metrics":{"Accuracy":"58.3","Parameters (Billions)":"70"},"uses_additional_data":true,"paper_date":"2024-02-15","paper":"/paper/openmathinstruct-1-a-1-8-million-math","paper_url":"https://arxiv.org/abs/2402.10176v2","paper_title":"OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset","code":"https://github.com/kipok/nemo-skills","n_code_links":1,"syntology":null},{"rank_in_archive_order":30,"model":"CR (GPT-4 model, w/o code)","metrics":{"Accuracy":"58.0"},"uses_additional_data":false,"paper_date":"2023-08-08","paper":"/paper/cumulative-reasoning-with-large-language","paper_url":"https://arxiv.org/abs/2308.04371v6","paper_title":"Cumulative Reasoning with Large Language Models","code":"https://github.com/iiis-ai/cumulative-reasoning","n_code_links":1,"syntology":{"n_ran":5,"n_unverified":1,"n_samples":6,"n_pointer_only_licence":6}},{"rank_in_archive_order":31,"model":"OpenMath-CodeLlama-13B (w/ code, SC, k=50)","metrics":{"Accuracy":"57.6","Parameters (Billions)":"13"},"uses_additional_data":true,"paper_date":"2024-02-15","paper":"/paper/openmathinstruct-1-a-1-8-million-math","paper_url":"https://arxiv.org/abs/2402.10176v2","paper_title":"OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset","code":"https://github.com/kipok/nemo-skills","n_code_links":1,"syntology":null},{"rank_in_archive_order":32,"model":"OpenMath-Mistral-7B (w/ code, SC, k=50)","metrics":{"Accuracy":"57.2","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":"2024-02-15","paper":"/paper/openmathinstruct-1-a-1-8-million-math","paper_url":"https://arxiv.org/abs/2402.10176v2","paper_title":"OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset","code":"https://github.com/kipok/nemo-skills","n_code_links":1,"syntology":null},{"rank_in_archive_order":33,"model":"ToRA 70B (w/ code, SC, k=50)","metrics":{"Accuracy":"56.9","Parameters (Billions)":"70"},"uses_additional_data":true,"paper_date":"2023-09-29","paper":"/paper/tora-a-tool-integrated-reasoning-agent-for","paper_url":"https://arxiv.org/abs/2309.17452v4","paper_title":"ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving","code":"https://github.com/microsoft/tora","n_code_links":1,"syntology":{"n_ran":5,"n_unverified":7,"n_samples":12,"n_pointer_only_licence":0}},{"rank_in_archive_order":34,"model":"SKiC (GPT-4 model)","metrics":{"Accuracy":"56.4"},"uses_additional_data":false,"paper_date":"2023-08-01","paper":"/paper/skills-in-context-prompting-unlocking","paper_url":"https://arxiv.org/abs/2308.00304v3","paper_title":"Skills-in-Context Prompting: Unlocking Compositionality in Large Language Models","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":35,"model":"DART-Math-Llama3-70B-Prop2Diff (0-shot CoT, w/o code)","metrics":{"Accuracy":"56.1","Parameters (Billions)":"70"},"uses_additional_data":true,"paper_date":"2024-06-18","paper":"/paper/dart-math-difficulty-aware-rejection-tuning-1","paper_url":"https://arxiv.org/abs/2407.13690v2","paper_title":"DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving","code":"https://github.com/hkust-nlp/dart-math","n_code_links":1,"syntology":{"n_ran":10,"n_unverified":2,"n_samples":12,"n_pointer_only_licence":0}},{"rank_in_archive_order":36,"model":"OpenMath-CodeLlama-7B (w/ code, SC, k=50)","metrics":{"Accuracy":"55.6","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":"2024-02-15","paper":"/paper/openmathinstruct-1-a-1-8-million-math","paper_url":"https://arxiv.org/abs/2402.10176v2","paper_title":"OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset","code":"https://github.com/kipok/nemo-skills","n_code_links":1,"syntology":null},{"rank_in_archive_order":37,"model":"MMOS-DeepSeekMath-7B(0-shot)","metrics":{"Accuracy":"55.0","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":"2024-02-23","paper":"/paper/an-empirical-study-of-data-ability-boundary","paper_url":"https://arxiv.org/abs/2403.00799v1","paper_title":"An Empirical Study of Data Ability Boundary in LLMs' Math Reasoning","code":"https://github.com/cyzhh/MMOS","n_code_links":1,"syntology":{"n_ran":9,"n_unverified":2,"n_samples":11,"n_pointer_only_licence":11}},{"rank_in_archive_order":38,"model":"DART-Math-Llama3-70B-Uniform (0-shot CoT, w/o code)","metrics":{"Accuracy":"54.9","Parameters (Billions)":"70"},"uses_additional_data":true,"paper_date":"2024-06-18","paper":"/paper/dart-math-difficulty-aware-rejection-tuning-1","paper_url":"https://arxiv.org/abs/2407.13690v2","paper_title":"DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving","code":"https://github.com/hkust-nlp/dart-math","n_code_links":1,"syntology":{"n_ran":10,"n_unverified":2,"n_samples":12,"n_pointer_only_licence":0}},{"rank_in_archive_order":39,"model":"PHP (GPT-4 model)","metrics":{"Accuracy":"53.9"},"uses_additional_data":false,"paper_date":"2023-04-19","paper":"/paper/progressive-hint-prompting-improves-reasoning","paper_url":"https://arxiv.org/abs/2304.09797v6","paper_title":"Progressive-Hint Prompting Improves Reasoning in Large Language Models","code":"https://github.com/chuanyang-Zheng/Progressive-Hint","n_code_links":1,"syntology":{"n_ran":3,"n_unverified":1,"n_samples":4,"n_pointer_only_licence":4}},{"rank_in_archive_order":40,"model":"DART-Math-DSMath-7B-Prop2Diff (0-shot CoT, w/o code)","metrics":{"Accuracy":"53.6","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":"2024-06-18","paper":"/paper/dart-math-difficulty-aware-rejection-tuning-1","paper_url":"https://arxiv.org/abs/2407.13690v2","paper_title":"DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving","code":"https://github.com/hkust-nlp/dart-math","n_code_links":1,"syntology":{"n_ran":10,"n_unverified":2,"n_samples":12,"n_pointer_only_licence":0}},{"rank_in_archive_order":41,"model":"Gemini Ultra (4-shot)","metrics":{"Accuracy":"53.2"},"uses_additional_data":false,"paper_date":"2023-12-19","paper":"/paper/gemini-a-family-of-highly-capable-multimodal-1","paper_url":"https://arxiv.org/abs/2312.11805v5","paper_title":"Gemini: A Family of Highly Capable Multimodal Models","code":"https://github.com/valdecy/pybibx","n_code_links":1,"syntology":null},{"rank_in_archive_order":42,"model":"DART-Math-DSMath-7B-Uniform (0-shot CoT, w/o code)","metrics":{"Accuracy":"52.9","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":"2024-06-18","paper":"/paper/dart-math-difficulty-aware-rejection-tuning-1","paper_url":"https://arxiv.org/abs/2407.13690v2","paper_title":"DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving","code":"https://github.com/hkust-nlp/dart-math","n_code_links":1,"syntology":{"n_ran":10,"n_unverified":2,"n_samples":12,"n_pointer_only_licence":0}},{"rank_in_archive_order":43,"model":"GPT-4 model (w/ code, PAL)","metrics":{"Accuracy":"51.8"},"uses_additional_data":false,"paper_date":"2022-11-18","paper":"/paper/pal-program-aided-language-models","paper_url":"https://arxiv.org/abs/2211.10435v2","paper_title":"PAL: Program-aided Language Models","code":"https://github.com/srush/minichain","n_code_links":3,"syntology":{"n_ran":3,"n_unverified":0,"n_samples":3,"n_pointer_only_licence":0}},{"rank_in_archive_order":44,"model":"DeepSeekMATH-RL-7B (greedy decoding)","metrics":{"Accuracy":"51.7","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":"2024-02-05","paper":"/paper/deepseekmath-pushing-the-limits-of","paper_url":"https://arxiv.org/abs/2402.03300v3","paper_title":"DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models","code":"https://github.com/shibing624/medicalgpt","n_code_links":5,"syntology":{"n_ran":8,"n_unverified":16,"n_samples":24,"n_pointer_only_licence":0}},{"rank_in_archive_order":45,"model":"AlphaLLM (with MCTS)","metrics":{"Accuracy":"51"},"uses_additional_data":false,"paper_date":"2024-04-18","paper":"/paper/toward-self-improvement-of-llms-via","paper_url":"https://arxiv.org/abs/2404.12253v2","paper_title":"Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing","code":"https://github.com/yetianjhu/alphallm","n_code_links":1,"syntology":{"n_ran":13,"n_unverified":5,"n_samples":18,"n_pointer_only_licence":18}},{"rank_in_archive_order":46,"model":"ToRA-Code 34B (w/ code)","metrics":{"Accuracy":"50.8","Parameters (Billions)":"34"},"uses_additional_data":true,"paper_date":"2023-09-29","paper":"/paper/tora-a-tool-integrated-reasoning-agent-for","paper_url":"https://arxiv.org/abs/2309.17452v4","paper_title":"ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving","code":"https://github.com/microsoft/tora","n_code_links":1,"syntology":{"n_ran":5,"n_unverified":7,"n_samples":12,"n_pointer_only_licence":0}},{"rank_in_archive_order":47,"model":"OpenMath-CodeLlama-70B (w/ code)","metrics":{"Accuracy":"50.7","Parameters (Billions)":"70"},"uses_additional_data":true,"paper_date":"2024-02-15","paper":"/paper/openmathinstruct-1-a-1-8-million-math","paper_url":"https://arxiv.org/abs/2402.10176v2","paper_title":"OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset","code":"https://github.com/kipok/nemo-skills","n_code_links":1,"syntology":null},{"rank_in_archive_order":48,"model":"Minerva 540B (maj1@k, k=64)","metrics":{"Accuracy":"50.3"},"uses_additional_data":false,"paper_date":"2022-06-29","paper":"/paper/solving-quantitative-reasoning-problems-with","paper_url":"https://arxiv.org/abs/2206.14858v2","paper_title":"Solving Quantitative Reasoning Problems with Language Models","code":"https://github.com/gair-nlp/abel","n_code_links":1,"syntology":null},{"rank_in_archive_order":49,"model":"ToRA 70B (w/ code)","metrics":{"Accuracy":"49.7","Parameters (Billions)":"70"},"uses_additional_data":true,"paper_date":"2023-09-29","paper":"/paper/tora-a-tool-integrated-reasoning-agent-for","paper_url":"https://arxiv.org/abs/2309.17452v4","paper_title":"ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving","code":"https://github.com/microsoft/tora","n_code_links":1,"syntology":{"n_ran":5,"n_unverified":7,"n_samples":12,"n_pointer_only_licence":0}},{"rank_in_archive_order":50,"model":"MMOS-CODE-34B(0-shot)","metrics":{"Accuracy":"49.5","Parameters (Billions)":"34"},"uses_additional_data":true,"paper_date":"2024-02-23","paper":"/paper/an-empirical-study-of-data-ability-boundary","paper_url":"https://arxiv.org/abs/2403.00799v1","paper_title":"An Empirical Study of Data Ability Boundary in LLMs' Math Reasoning","code":"https://github.com/cyzhh/MMOS","n_code_links":1,"syntology":{"n_ran":9,"n_unverified":2,"n_samples":11,"n_pointer_only_licence":11}},{"rank_in_archive_order":51,"model":"DeepSeekMath-7B-KPMath-Plus","metrics":{"Accuracy":"48.8","Parameters (Billions)":"7"},"uses_additional_data":false,"paper_date":"2024-03-04","paper":"/paper/key-point-driven-data-synthesis-with-its","paper_url":"https://arxiv.org/abs/2403.02333v3","paper_title":"Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoning","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":52,"model":"PaLM 2 (few-shot, k=4, SC)","metrics":{"Accuracy":"48.8"},"uses_additional_data":false,"paper_date":"2023-05-17","paper":"/paper/palm-2-technical-report-1","paper_url":"https://arxiv.org/abs/2305.10403v3","paper_title":"PaLM 2 Technical Report","code":"https://github.com/eternityyw/tram-benchmark","n_code_links":1,"syntology":null},{"rank_in_archive_order":53,"model":"Llemma-34B-KPMath-Plus","metrics":{"Accuracy":"48.6","Parameters (Billions)":"34"},"uses_additional_data":false,"paper_date":"2024-03-04","paper":"/paper/key-point-driven-data-synthesis-with-its","paper_url":"https://arxiv.org/abs/2403.02333v3","paper_title":"Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoning","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":54,"model":"OpenMath-CodeLlama-34B (w/ code)","metrics":{"Accuracy":"48.3","Parameters (Billions)":"34"},"uses_additional_data":true,"paper_date":"2024-02-15","paper":"/paper/openmathinstruct-1-a-1-8-million-math","paper_url":"https://arxiv.org/abs/2402.10176v2","paper_title":"OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset","code":"https://github.com/kipok/nemo-skills","n_code_links":1,"syntology":null},{"rank_in_archive_order":55,"model":"Shepherd + DeepSeek-67B (SFT on MetaMATH + PRM rerank, k=256)","metrics":{"Accuracy":"48.1","Parameters (Billions)":"67"},"uses_additional_data":true,"paper_date":"2023-12-14","paper":"/paper/math-shepherd-a-label-free-step-by-step","paper_url":"https://arxiv.org/abs/2312.08935v3","paper_title":"Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations","code":"https://github.com/hkust-nlp/b-star","n_code_links":3,"syntology":null},{"rank_in_archive_order":56,"model":"ToRA-Code 13B (w/ code)","metrics":{"Accuracy":"48.1","Parameters (Billions)":"13"},"uses_additional_data":true,"paper_date":"2023-09-29","paper":"/paper/tora-a-tool-integrated-reasoning-agent-for","paper_url":"https://arxiv.org/abs/2309.17452v4","paper_title":"ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving","code":"https://github.com/microsoft/tora","n_code_links":1,"syntology":{"n_ran":5,"n_unverified":7,"n_samples":12,"n_pointer_only_licence":0}},{"rank_in_archive_order":57,"model":"Minerva 8B (maj5@256)","metrics":{"Accuracy":"47.6","Parameters (Billions)":"8"},"uses_additional_data":false,"paper_date":"2022-06-29","paper":"/paper/solving-quantitative-reasoning-problems-with","paper_url":"https://arxiv.org/abs/2206.14858v2","paper_title":"Solving Quantitative Reasoning Problems with Language Models","code":"https://github.com/gair-nlp/abel","n_code_links":1,"syntology":null},{"rank_in_archive_order":58,"model":"Mistral-7B-KPMath-Plus","metrics":{"Accuracy":"46.8","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":"2024-03-04","paper":"/paper/key-point-driven-data-synthesis-with-its","paper_url":"https://arxiv.org/abs/2403.02333v3","paper_title":"Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoning","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":59,"model":"DART-Math-Llama3-8B-Prop2Diff (0-shot CoT, w/o code)","metrics":{"Accuracy":"46.6","Parameters (Billions)":"8"},"uses_additional_data":true,"paper_date":"2024-06-18","paper":"/paper/dart-math-difficulty-aware-rejection-tuning-1","paper_url":"https://arxiv.org/abs/2407.13690v2","paper_title":"DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving","code":"https://github.com/hkust-nlp/dart-math","n_code_links":1,"syntology":{"n_ran":10,"n_unverified":2,"n_samples":12,"n_pointer_only_licence":0}},{"rank_in_archive_order":60,"model":"OpenMath-Llama2-70B (w/ code)","metrics":{"Accuracy":"46.3","Parameters (Billions)":"70"},"uses_additional_data":true,"paper_date":"2024-02-15","paper":"/paper/openmathinstruct-1-a-1-8-million-math","paper_url":"https://arxiv.org/abs/2402.10176v2","paper_title":"OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset","code":"https://github.com/kipok/nemo-skills","n_code_links":1,"syntology":null},{"rank_in_archive_order":61,"model":"OpenMath-CodeLlama-13B (w/ code)","metrics":{"Accuracy":"45.5","Parameters (Billions)":"13"},"uses_additional_data":true,"paper_date":"2024-02-15","paper":"/paper/openmathinstruct-1-a-1-8-million-math","paper_url":"https://arxiv.org/abs/2402.10176v2","paper_title":"OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset","code":"https://github.com/kipok/nemo-skills","n_code_links":1,"syntology":null},{"rank_in_archive_order":62,"model":"DART-Math-Mistral-7B-Prop2Diff (0-shot CoT, w/o code)","metrics":{"Accuracy":"45.5","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":"2024-06-18","paper":"/paper/dart-math-difficulty-aware-rejection-tuning-1","paper_url":"https://arxiv.org/abs/2407.13690v2","paper_title":"DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving","code":"https://github.com/hkust-nlp/dart-math","n_code_links":1,"syntology":{"n_ran":10,"n_unverified":2,"n_samples":12,"n_pointer_only_licence":0}},{"rank_in_archive_order":63,"model":"DART-Math-Llama3-8B-Uniform (0-shot CoT, w/o code)","metrics":{"Accuracy":"45.3","Parameters (Billions)":"8"},"uses_additional_data":true,"paper_date":"2024-06-18","paper":"/paper/dart-math-difficulty-aware-rejection-tuning-1","paper_url":"https://arxiv.org/abs/2407.13690v2","paper_title":"DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving","code":"https://github.com/hkust-nlp/dart-math","n_code_links":1,"syntology":{"n_ran":10,"n_unverified":2,"n_samples":12,"n_pointer_only_licence":0}},{"rank_in_archive_order":64,"model":"MathCoder-CL-34B","metrics":{"Accuracy":"45.2","Parameters (Billions)":"34"},"uses_additional_data":true,"paper_date":"2023-10-05","paper":"/paper/mathcoder-seamless-code-integration-in-llms","paper_url":"https://arxiv.org/abs/2310.03731v1","paper_title":"MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning","code":"https://github.com/mathllm/mathcoder","n_code_links":1,"syntology":{"n_ran":2,"n_unverified":0,"n_samples":2,"n_pointer_only_licence":0}},{"rank_in_archive_order":65,"model":"MathCoder-L-34B","metrics":{"Accuracy":"45.1","Parameters (Billions)":"34"},"uses_additional_data":true,"paper_date":"2023-10-05","paper":"/paper/mathcoder-seamless-code-integration-in-llms","paper_url":"https://arxiv.org/abs/2310.03731v1","paper_title":"MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning","code":"https://github.com/mathllm/mathcoder","n_code_links":1,"syntology":{"n_ran":2,"n_unverified":0,"n_samples":2,"n_pointer_only_licence":0}},{"rank_in_archive_order":66,"model":"MMIQC-72B","metrics":{"Accuracy":"45.0","Parameters (Billions)":"72"},"uses_additional_data":true,"paper_date":"2024-01-17","paper":"/paper/augmenting-math-word-problems-via-iterative","paper_url":"https://arxiv.org/abs/2401.09003v5","paper_title":"Augmenting Math Word Problems via Iterative Question Composing","code":"https://github.com/iiis-ai/iterativequestioncomposing","n_code_links":1,"syntology":{"n_ran":4,"n_unverified":0,"n_samples":4,"n_pointer_only_licence":4}},{"rank_in_archive_order":67,"model":"ToRA-Code 7B (w/ code)","metrics":{"Accuracy":"44.6","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":"2023-09-29","paper":"/paper/tora-a-tool-integrated-reasoning-agent-for","paper_url":"https://arxiv.org/abs/2309.17452v4","paper_title":"ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving","code":"https://github.com/microsoft/tora","n_code_links":1,"syntology":{"n_ran":5,"n_unverified":7,"n_samples":12,"n_pointer_only_licence":0}},{"rank_in_archive_order":68,"model":"OpenMath-Mistral-7B (w/ code)","metrics":{"Accuracy":"44.5","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":"2024-02-15","paper":"/paper/openmathinstruct-1-a-1-8-million-math","paper_url":"https://arxiv.org/abs/2402.10176v2","paper_title":"OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset","code":"https://github.com/kipok/nemo-skills","n_code_links":1,"syntology":null},{"rank_in_archive_order":69,"model":"MMOS-CODE-7B(0-shot)","metrics":{"Accuracy":"44.3","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":"2024-02-23","paper":"/paper/an-empirical-study-of-data-ability-boundary","paper_url":"https://arxiv.org/abs/2403.00799v1","paper_title":"An Empirical Study of Data Ability Boundary in LLMs' Math Reasoning","code":"https://github.com/cyzhh/MMOS","n_code_links":1,"syntology":{"n_ran":9,"n_unverified":2,"n_samples":11,"n_pointer_only_licence":11}},{"rank_in_archive_order":70,"model":"OpenMath-CodeLlama-7B (w/ code)","metrics":{"Accuracy":"43.6","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":"2024-02-15","paper":"/paper/openmathinstruct-1-a-1-8-million-math","paper_url":"https://arxiv.org/abs/2402.10176v2","paper_title":"OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset","code":"https://github.com/kipok/nemo-skills","n_code_links":1,"syntology":null},{"rank_in_archive_order":71,"model":"Shepherd+Mistral-7B (SFT on MetaMATH + PRM RL+ PRM rerank, k=256)","metrics":{"Accuracy":"43.5","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":"2023-12-14","paper":"/paper/math-shepherd-a-label-free-step-by-step","paper_url":"https://arxiv.org/abs/2312.08935v3","paper_title":"Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations","code":"https://github.com/hkust-nlp/b-star","n_code_links":3,"syntology":null},{"rank_in_archive_order":72,"model":"DART-Math-Mistral-7B-Uniform (0-shot CoT, w/o code)","metrics":{"Accuracy":"43.5","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":"2024-06-18","paper":"/paper/dart-math-difficulty-aware-rejection-tuning-1","paper_url":"https://arxiv.org/abs/2407.13690v2","paper_title":"DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving","code":"https://github.com/hkust-nlp/dart-math","n_code_links":1,"syntology":{"n_ran":10,"n_unverified":2,"n_samples":12,"n_pointer_only_licence":0}},{"rank_in_archive_order":73,"model":"Minerva 62B (maj1@k, k=64)","metrics":{"Accuracy":"43.4","Parameters (Billions)":"62"},"uses_additional_data":false,"paper_date":"2022-06-29","paper":"/paper/solving-quantitative-reasoning-problems-with","paper_url":"https://arxiv.org/abs/2206.14858v2","paper_title":"Solving Quantitative Reasoning Problems with Language Models","code":"https://github.com/gair-nlp/abel","n_code_links":1,"syntology":null},{"rank_in_archive_order":74,"model":"ToRA 13B (w/ code)","metrics":{"Accuracy":"43.0","Parameters (Billions)":"13"},"uses_additional_data":true,"paper_date":"2023-09-29","paper":"/paper/tora-a-tool-integrated-reasoning-agent-for","paper_url":"https://arxiv.org/abs/2309.17452v4","paper_title":"ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving","code":"https://github.com/microsoft/tora","n_code_links":1,"syntology":{"n_ran":5,"n_unverified":7,"n_samples":12,"n_pointer_only_licence":0}},{"rank_in_archive_order":75,"model":"GPT-4","metrics":{"Accuracy":"42.5"},"uses_additional_data":false,"paper_date":"2023-03-22","paper":"/paper/sparks-of-artificial-general-intelligence","paper_url":"https://arxiv.org/abs/2303.12712v5","paper_title":"Sparks of Artificial General Intelligence: Early experiments with GPT-4","code":"https://github.com/microsoft/guidance","n_code_links":3,"syntology":{"n_ran":0,"n_unverified":10,"n_samples":10,"n_pointer_only_licence":0}},{"rank_in_archive_order":76,"model":"SFT-Mistral-7B","metrics":{"Accuracy":"41.8","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":null,"paper":null,"paper_url":null,"paper_title":"","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":77,"model":"Llama2-13B-KPMath-Plus","metrics":{"Accuracy":"41","Parameters (Billions)":"13"},"uses_additional_data":false,"paper_date":"2024-03-04","paper":"/paper/key-point-driven-data-synthesis-with-its","paper_url":"https://arxiv.org/abs/2403.02333v3","paper_title":"Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoning","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":78,"model":"ToRA 7B (w/ code)","metrics":{"Accuracy":"40.1","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":"2023-09-29","paper":"/paper/tora-a-tool-integrated-reasoning-agent-for","paper_url":"https://arxiv.org/abs/2309.17452v4","paper_title":"ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving","code":"https://github.com/microsoft/tora","n_code_links":1,"syntology":{"n_ran":5,"n_unverified":7,"n_samples":12,"n_pointer_only_licence":0}},{"rank_in_archive_order":79,"model":"MathCoder-CL-13B","metrics":{"Accuracy":"35.9","Parameters (Billions)":"13"},"uses_additional_data":true,"paper_date":"2023-10-05","paper":"/paper/mathcoder-seamless-code-integration-in-llms","paper_url":"https://arxiv.org/abs/2310.03731v1","paper_title":"MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning","code":"https://github.com/mathllm/mathcoder","n_code_links":1,"syntology":{"n_ran":2,"n_unverified":0,"n_samples":2,"n_pointer_only_licence":0}},{"rank_in_archive_order":80,"model":"MuggleMATH-70B","metrics":{"Accuracy":"35.6","Parameters (Billions)":"70"},"uses_additional_data":true,"paper_date":"2023-10-09","paper":"/paper/query-and-response-augmentation-cannot-help","paper_url":"https://arxiv.org/abs/2310.05506v3","paper_title":"MuggleMath: Assessing the Impact of Query and Response Augmentation on Math Reasoning","code":"https://github.com/ofa-sys/gsm8k-screl","n_code_links":1,"syntology":null},{"rank_in_archive_order":81,"model":"PaLM 2 (few-shot, k=4, CoT)","metrics":{"Accuracy":"34.3"},"uses_additional_data":false,"paper_date":"2023-05-17","paper":"/paper/palm-2-technical-report-1","paper_url":"https://arxiv.org/abs/2305.10403v3","paper_title":"PaLM 2 Technical Report","code":"https://github.com/eternityyw/tram-benchmark","n_code_links":1,"syntology":null},{"rank_in_archive_order":82,"model":"Minerva 540B","metrics":{"Accuracy":"33.6","Parameters (Billions)":"540"},"uses_additional_data":false,"paper_date":"2022-06-29","paper":"/paper/solving-quantitative-reasoning-problems-with","paper_url":"https://arxiv.org/abs/2206.14858v2","paper_title":"Solving Quantitative Reasoning Problems with Language Models","code":"https://github.com/gair-nlp/abel","n_code_links":1,"syntology":null},{"rank_in_archive_order":83,"model":"Minerva 540B (5-shot) mCoT","metrics":{"Accuracy":"33.6","Parameters (Billions)":"540"},"uses_additional_data":false,"paper_date":"2022-11-16","paper":"/paper/galactica-a-large-language-model-for-science-1","paper_url":"https://arxiv.org/abs/2211.09085v1","paper_title":"Galactica: A Large Language Model for Science","code":"https://github.com/paperswithcode/galai","n_code_links":1,"syntology":{"n_ran":0,"n_unverified":2,"n_samples":2,"n_pointer_only_licence":0}},{"rank_in_archive_order":84,"model":"Shepherd + Mistral-7B (SFT on MetaMATH + PRM RL)","metrics":{"Accuracy":"33.0","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":"2023-12-14","paper":"/paper/math-shepherd-a-label-free-step-by-step","paper_url":"https://arxiv.org/abs/2312.08935v3","paper_title":"Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations","code":"https://github.com/hkust-nlp/b-star","n_code_links":3,"syntology":null},{"rank_in_archive_order":85,"model":"WizardMath-7B-V1.1","metrics":{"Accuracy":"33.0","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":"2023-08-18","paper":"/paper/wizardmath-empowering-mathematical-reasoning","paper_url":"https://arxiv.org/abs/2308.09583v2","paper_title":"WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct","code":"https://github.com/nlpxucan/wizardlm","n_code_links":1,"syntology":{"n_ran":10,"n_unverified":6,"n_samples":16,"n_pointer_only_licence":16}},{"rank_in_archive_order":86,"model":"Gemini Pro (4-shot)","metrics":{"Accuracy":"32.6"},"uses_additional_data":false,"paper_date":"2023-12-19","paper":"/paper/gemini-a-family-of-highly-capable-multimodal-1","paper_url":"https://arxiv.org/abs/2312.11805v5","paper_title":"Gemini: A Family of Highly Capable Multimodal Models","code":"https://github.com/valdecy/pybibx","n_code_links":1,"syntology":null},{"rank_in_archive_order":87,"model":"MuggleMATH-13B","metrics":{"Accuracy":"30.7","Parameters (Billions)":"13"},"uses_additional_data":true,"paper_date":"2023-10-09","paper":"/paper/query-and-response-augmentation-cannot-help","paper_url":"https://arxiv.org/abs/2310.05506v3","paper_title":"MuggleMath: Assessing the Impact of Query and Response Augmentation on Math Reasoning","code":"https://github.com/ofa-sys/gsm8k-screl","n_code_links":1,"syntology":null},{"rank_in_archive_order":88,"model":"MathCoder-CL-7B","metrics":{"Accuracy":"30.2","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":"2023-10-05","paper":"/paper/mathcoder-seamless-code-integration-in-llms","paper_url":"https://arxiv.org/abs/2310.03731v1","paper_title":"MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning","code":"https://github.com/mathllm/mathcoder","n_code_links":1,"syntology":{"n_ran":2,"n_unverified":0,"n_samples":2,"n_pointer_only_licence":0}},{"rank_in_archive_order":89,"model":"MathCoder-L-13B","metrics":{"Accuracy":"29.9","Parameters (Billions)":"13"},"uses_additional_data":true,"paper_date":"2023-10-05","paper":"/paper/mathcoder-seamless-code-integration-in-llms","paper_url":"https://arxiv.org/abs/2310.03731v1","paper_title":"MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning","code":"https://github.com/mathllm/mathcoder","n_code_links":1,"syntology":{"n_ran":2,"n_unverified":0,"n_samples":2,"n_pointer_only_licence":0}},{"rank_in_archive_order":90,"model":"Qwen2idae-16x14B (4-shot)","metrics":{"Accuracy":"29.9"},"uses_additional_data":false,"paper_date":"2024-01-05","paper":"/paper/parameter-efficient-sparsity-crafting-from","paper_url":"https://arxiv.org/abs/2401.02731v4","paper_title":"Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks","code":"https://github.com/wuhy68/parameter-efficient-moe","n_code_links":2,"syntology":null},{"rank_in_archive_order":91,"model":"OpenChat-3.5-1210 7B","metrics":{"Accuracy":"28.9","Parameters (Billions)":"7"},"uses_additional_data":false,"paper_date":"2023-09-20","paper":"/paper/openchat-advancing-open-source-language","paper_url":"https://arxiv.org/abs/2309.11235v2","paper_title":"OpenChat: Advancing Open-source Language Models with Mixed-Quality Data","code":"https://github.com/imoneoi/openchat","n_code_links":1,"syntology":{"n_ran":0,"n_unverified":1,"n_samples":1,"n_pointer_only_licence":0}},{"rank_in_archive_order":92,"model":"OpenChat-3.5 7B","metrics":{"Accuracy":"28.6","Parameters (Billions)":"7"},"uses_additional_data":false,"paper_date":"2023-09-20","paper":"/paper/openchat-advancing-open-source-language","paper_url":"https://arxiv.org/abs/2309.11235v2","paper_title":"OpenChat: Advancing Open-source Language Models with Mixed-Quality Data","code":"https://github.com/imoneoi/openchat","n_code_links":1,"syntology":{"n_ran":0,"n_unverified":1,"n_samples":1,"n_pointer_only_licence":0}},{"rank_in_archive_order":93,"model":"Mixtral 8x7B (maj@4)","metrics":{"Accuracy":"28.4"},"uses_additional_data":false,"paper_date":"2024-01-08","paper":"/paper/mixtral-of-experts","paper_url":"https://arxiv.org/abs/2401.04088v1","paper_title":"Mixtral of Experts","code":"https://github.com/jingyaogong/minimind","n_code_links":6,"syntology":{"n_ran":5,"n_unverified":0,"n_samples":5,"n_pointer_only_licence":0}},{"rank_in_archive_order":94,"model":"Minerva 62B (4-shot)","metrics":{"Accuracy":"27.6","Parameters (Billions)":"62"},"uses_additional_data":false,"paper_date":"2022-06-29","paper":"/paper/solving-quantitative-reasoning-problems-with","paper_url":"https://arxiv.org/abs/2206.14858v2","paper_title":"Solving Quantitative Reasoning Problems with Language Models","code":"https://github.com/gair-nlp/abel","n_code_links":1,"syntology":null},{"rank_in_archive_order":95,"model":"MetaMath 70B","metrics":{"Accuracy":"26.0","Parameters (Billions)":"70"},"uses_additional_data":true,"paper_date":"2023-09-21","paper":"/paper/metamath-bootstrap-your-own-mathematical","paper_url":"https://arxiv.org/abs/2309.12284v4","paper_title":"MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models","code":"https://github.com/meta-math/MetaMath","n_code_links":1,"syntology":{"n_ran":15,"n_unverified":7,"n_samples":22,"n_pointer_only_licence":0}},{"rank_in_archive_order":96,"model":"MuggleMATH 7B","metrics":{"Accuracy":"25.8","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":"2023-10-09","paper":"/paper/query-and-response-augmentation-cannot-help","paper_url":"https://arxiv.org/abs/2310.05506v3","paper_title":"MuggleMath: Assessing the Impact of Query and Response Augmentation on Math Reasoning","code":"https://github.com/ofa-sys/gsm8k-screl","n_code_links":1,"syntology":null},{"rank_in_archive_order":97,"model":"Minerva 8B (maj1@k, k=64)","metrics":{"Accuracy":"25.4","Parameters (Billions)":"8"},"uses_additional_data":false,"paper_date":"2022-06-29","paper":"/paper/solving-quantitative-reasoning-problems-with","paper_url":"https://arxiv.org/abs/2206.14858v2","paper_title":"Solving Quantitative Reasoning Problems with Language Models","code":"https://github.com/gair-nlp/abel","n_code_links":1,"syntology":null},{"rank_in_archive_order":98,"model":"MathCoder-L-7B","metrics":{"Accuracy":"23.3","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":"2023-10-05","paper":"/paper/mathcoder-seamless-code-integration-in-llms","paper_url":"https://arxiv.org/abs/2310.03731v1","paper_title":"MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning","code":"https://github.com/mathllm/mathcoder","n_code_links":1,"syntology":{"n_ran":2,"n_unverified":0,"n_samples":2,"n_pointer_only_licence":0}},{"rank_in_archive_order":99,"model":"WizardMath-70B-V1.0","metrics":{"Accuracy":"22.7","Parameters (Billions)":"70"},"uses_additional_data":true,"paper_date":"2023-08-18","paper":"/paper/wizardmath-empowering-mathematical-reasoning","paper_url":"https://arxiv.org/abs/2308.09583v2","paper_title":"WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct","code":"https://github.com/nlpxucan/wizardlm","n_code_links":1,"syntology":{"n_ran":10,"n_unverified":6,"n_samples":16,"n_pointer_only_licence":16}},{"rank_in_archive_order":100,"model":"Camelidae-8×34B (4-shot)","metrics":{"Accuracy":"22.6"},"uses_additional_data":false,"paper_date":"2024-01-05","paper":"/paper/parameter-efficient-sparsity-crafting-from","paper_url":"https://arxiv.org/abs/2401.02731v4","paper_title":"Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks","code":"https://github.com/wuhy68/parameter-efficient-moe","n_code_links":2,"syntology":null},{"rank_in_archive_order":101,"model":"MetaMath 13B","metrics":{"Accuracy":"22.5","Parameters (Billions)":"13"},"uses_additional_data":true,"paper_date":"2023-09-21","paper":"/paper/metamath-bootstrap-your-own-mathematical","paper_url":"https://arxiv.org/abs/2309.12284v4","paper_title":"MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models","code":"https://github.com/meta-math/MetaMath","n_code_links":1,"syntology":{"n_ran":15,"n_unverified":7,"n_samples":22,"n_pointer_only_licence":0}},{"rank_in_archive_order":102,"model":"LLaMA 65B (maj1@k)","metrics":{"Accuracy":"20.5","Parameters (Billions)":"65"},"uses_additional_data":false,"paper_date":"2023-02-27","paper":"/paper/llama-open-and-efficient-foundation-language-1","paper_url":"https://arxiv.org/abs/2302.13971v1","paper_title":"LLaMA: Open and Efficient Foundation Language Models","code":"https://github.com/huggingface/transformers","n_code_links":57,"syntology":{"n_ran":26,"n_unverified":32,"n_samples":58,"n_pointer_only_licence":4}},{"rank_in_archive_order":103,"model":"GAL 120B (5-shot) mCoT","metrics":{"Accuracy":"20.4","Parameters (Billions)":"120"},"uses_additional_data":false,"paper_date":"2022-11-16","paper":"/paper/galactica-a-large-language-model-for-science-1","paper_url":"https://arxiv.org/abs/2211.09085v1","paper_title":"Galactica: A Large Language Model for Science","code":"https://github.com/paperswithcode/galai","n_code_links":1,"syntology":{"n_ran":0,"n_unverified":2,"n_samples":2,"n_pointer_only_licence":0}},{"rank_in_archive_order":104,"model":"MetaMath 7B","metrics":{"Accuracy":"19.4","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":"2023-09-21","paper":"/paper/metamath-bootstrap-your-own-mathematical","paper_url":"https://arxiv.org/abs/2309.12284v4","paper_title":"MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models","code":"https://github.com/meta-math/MetaMath","n_code_links":1,"syntology":{"n_ran":15,"n_unverified":7,"n_samples":22,"n_pointer_only_licence":0}},{"rank_in_archive_order":105,"model":"davinci-002 175B","metrics":{"Accuracy":"19.1","Parameters (Billions)":"175"},"uses_additional_data":false,"paper_date":"2022-06-29","paper":"/paper/solving-quantitative-reasoning-problems-with","paper_url":"https://arxiv.org/abs/2206.14858v2","paper_title":"Solving Quantitative Reasoning Problems with Language Models","code":"https://github.com/gair-nlp/abel","n_code_links":1,"syntology":null},{"rank_in_archive_order":106,"model":"Branch-Train-MiX 4x7B (sampling top-2 experts)","metrics":{"Accuracy":"17.8"},"uses_additional_data":false,"paper_date":"2024-03-12","paper":"/paper/branch-train-mix-mixing-expert-llms-into-a","paper_url":"https://arxiv.org/abs/2403.07816v1","paper_title":"Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM","code":"https://github.com/Leeroo-AI/mergoo","n_code_links":1,"syntology":null},{"rank_in_archive_order":107,"model":"GAL 120B <work>","metrics":{"Accuracy":"16.6","Parameters (Billions)":"120"},"uses_additional_data":false,"paper_date":"2022-11-16","paper":"/paper/galactica-a-large-language-model-for-science-1","paper_url":"https://arxiv.org/abs/2211.09085v1","paper_title":"Galactica: A Large Language Model for Science","code":"https://github.com/paperswithcode/galai","n_code_links":1,"syntology":{"n_ran":0,"n_unverified":2,"n_samples":2,"n_pointer_only_licence":0}},{"rank_in_archive_order":108,"model":"LLaMA 33B-maj1@k","metrics":{"Accuracy":"15.2","Parameters (Billions)":"33"},"uses_additional_data":false,"paper_date":"2023-02-27","paper":"/paper/llama-open-and-efficient-foundation-language-1","paper_url":"https://arxiv.org/abs/2302.13971v1","paper_title":"LLaMA: Open and Efficient Foundation Language Models","code":"https://github.com/huggingface/transformers","n_code_links":57,"syntology":{"n_ran":26,"n_unverified":32,"n_samples":58,"n_pointer_only_licence":4}},{"rank_in_archive_order":109,"model":"Minerva 8B","metrics":{"Accuracy":"14.1","Parameters (Billions)":"8"},"uses_additional_data":false,"paper_date":"2022-06-29","paper":"/paper/solving-quantitative-reasoning-problems-with","paper_url":"https://arxiv.org/abs/2206.14858v2","paper_title":"Solving Quantitative Reasoning Problems with Language Models","code":"https://github.com/gair-nlp/abel","n_code_links":1,"syntology":null},{"rank_in_archive_order":110,"model":"WizardMath-13B-V1.0","metrics":{"Accuracy":"14.0","Parameters (Billions)":"13"},"uses_additional_data":true,"paper_date":"2023-08-18","paper":"/paper/wizardmath-empowering-mathematical-reasoning","paper_url":"https://arxiv.org/abs/2308.09583v2","paper_title":"WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct","code":"https://github.com/nlpxucan/wizardlm","n_code_links":1,"syntology":{"n_ran":10,"n_unverified":6,"n_samples":16,"n_pointer_only_licence":16}},{"rank_in_archive_order":111,"model":"Mistral 7B (maj@4)","metrics":{"Accuracy":"13.1","Parameters (Billions)":"7"},"uses_additional_data":false,"paper_date":"2023-10-10","paper":"/paper/mistral-7b","paper_url":"https://arxiv.org/abs/2310.06825v1","paper_title":"Mistral 7B","code":"https://github.com/mistralai/mistral-src","n_code_links":6,"syntology":{"n_ran":9,"n_unverified":2,"n_samples":11,"n_pointer_only_licence":1}},{"rank_in_archive_order":112,"model":"GAL 30B (5-shot) mCoT","metrics":{"Accuracy":"12.7","Parameters (Billions)":"30"},"uses_additional_data":false,"paper_date":"2022-11-16","paper":"/paper/galactica-a-large-language-model-for-science-1","paper_url":"https://arxiv.org/abs/2211.09085v1","paper_title":"Galactica: A Large Language Model for Science","code":"https://github.com/paperswithcode/galai","n_code_links":1,"syntology":{"n_ran":0,"n_unverified":2,"n_samples":2,"n_pointer_only_licence":0}},{"rank_in_archive_order":113,"model":"Mistral 7B (maj@4)","metrics":{"Accuracy":"12.7","Parameters (Billions)":"7"},"uses_additional_data":false,"paper_date":"2024-01-08","paper":"/paper/mixtral-of-experts","paper_url":"https://arxiv.org/abs/2401.04088v1","paper_title":"Mixtral of Experts","code":"https://github.com/jingyaogong/minimind","n_code_links":6,"syntology":{"n_ran":5,"n_unverified":0,"n_samples":5,"n_pointer_only_licence":0}},{"rank_in_archive_order":114,"model":"GAL 30B <work>","metrics":{"Accuracy":"11.4","Parameters (Billions)":"30"},"uses_additional_data":false,"paper_date":"2022-11-16","paper":"/paper/galactica-a-large-language-model-for-science-1","paper_url":"https://arxiv.org/abs/2211.09085v1","paper_title":"Galactica: A Large Language Model for Science","code":"https://github.com/paperswithcode/galai","n_code_links":1,"syntology":{"n_ran":0,"n_unverified":2,"n_samples":2,"n_pointer_only_licence":0}},{"rank_in_archive_order":115,"model":"WizardMath-7B-V1.0","metrics":{"Accuracy":"10.7","Parameters (Billions)":"7"},"uses_additional_data":true,"paper_date":"2023-08-18","paper":"/paper/wizardmath-empowering-mathematical-reasoning","paper_url":"https://arxiv.org/abs/2308.09583v2","paper_title":"WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct","code":"https://github.com/nlpxucan/wizardlm","n_code_links":1,"syntology":{"n_ran":10,"n_unverified":6,"n_samples":16,"n_pointer_only_licence":16}},{"rank_in_archive_order":116,"model":"LLaMA 65B","metrics":{"Accuracy":"10.6","Parameters (Billions)":"65"},"uses_additional_data":false,"paper_date":"2023-02-27","paper":"/paper/llama-open-and-efficient-foundation-language-1","paper_url":"https://arxiv.org/abs/2302.13971v1","paper_title":"LLaMA: Open and Efficient Foundation Language Models","code":"https://github.com/huggingface/transformers","n_code_links":57,"syntology":{"n_ran":26,"n_unverified":32,"n_samples":58,"n_pointer_only_licence":4}},{"rank_in_archive_order":117,"model":"PaLM 540B","metrics":{"Accuracy":"8.8","Parameters (Billions)":"540"},"uses_additional_data":false,"paper_date":"2022-06-29","paper":"/paper/solving-quantitative-reasoning-problems-with","paper_url":"https://arxiv.org/abs/2206.14858v2","paper_title":"Solving Quantitative Reasoning Problems with Language Models","code":"https://github.com/gair-nlp/abel","n_code_links":1,"syntology":null},{"rank_in_archive_order":118,"model":"PaLM 540B (5-shot) mCoT","metrics":{"Accuracy":"8.8","Parameters (Billions)":"540"},"uses_additional_data":false,"paper_date":"2022-11-16","paper":"/paper/galactica-a-large-language-model-for-science-1","paper_url":"https://arxiv.org/abs/2211.09085v1","paper_title":"Galactica: A Large Language Model for Science","code":"https://github.com/paperswithcode/galai","n_code_links":1,"syntology":{"n_ran":0,"n_unverified":2,"n_samples":2,"n_pointer_only_licence":0}},{"rank_in_archive_order":119,"model":"LLaMA 13B-maj1@k","metrics":{"Accuracy":"8.8","Parameters (Billions)":"13"},"uses_additional_data":false,"paper_date":"2023-02-27","paper":"/paper/llama-open-and-efficient-foundation-language-1","paper_url":"https://arxiv.org/abs/2302.13971v1","paper_title":"LLaMA: Open and Efficient Foundation Language Models","code":"https://github.com/huggingface/transformers","n_code_links":57,"syntology":{"n_ran":26,"n_unverified":32,"n_samples":58,"n_pointer_only_licence":4}},{"rank_in_archive_order":120,"model":"LLaMA 33B","metrics":{"Accuracy":"7.1","Parameters (Billions)":"33"},"uses_additional_data":false,"paper_date":"2023-02-27","paper":"/paper/llama-open-and-efficient-foundation-language-1","paper_url":"https://arxiv.org/abs/2302.13971v1","paper_title":"LLaMA: Open and Efficient Foundation Language Models","code":"https://github.com/huggingface/transformers","n_code_links":57,"syntology":{"n_ran":26,"n_unverified":32,"n_samples":58,"n_pointer_only_licence":4}},{"rank_in_archive_order":121,"model":"LLaMA 7B-maj1@k","metrics":{"Accuracy":"6.9","Parameters (Billions)":"7"},"uses_additional_data":false,"paper_date":"2023-02-27","paper":"/paper/llama-open-and-efficient-foundation-language-1","paper_url":"https://arxiv.org/abs/2302.13971v1","paper_title":"LLaMA: Open and Efficient Foundation Language Models","code":"https://github.com/huggingface/transformers","n_code_links":57,"syntology":{"n_ran":26,"n_unverified":32,"n_samples":58,"n_pointer_only_licence":4}},{"rank_in_archive_order":122,"model":"GPT-2 (1.5B)","metrics":{"Accuracy":"6.9","Parameters (Billions)":"1.5"},"uses_additional_data":false,"paper_date":"2021-03-05","paper":"/paper/measuring-mathematical-problem-solving-with","paper_url":"https://arxiv.org/abs/2103.03874v2","paper_title":"Measuring Mathematical Problem Solving With the MATH Dataset","code":"https://github.com/hendrycks/math","n_code_links":5,"syntology":{"n_ran":3,"n_unverified":0,"n_samples":3,"n_pointer_only_licence":1}},{"rank_in_archive_order":123,"model":"GPT-2 (0.7B)","metrics":{"Accuracy":"6.4","Parameters (Billions)":"0.7"},"uses_additional_data":false,"paper_date":"2021-03-05","paper":"/paper/measuring-mathematical-problem-solving-with","paper_url":"https://arxiv.org/abs/2103.03874v2","paper_title":"Measuring Mathematical Problem Solving With the MATH Dataset","code":"https://github.com/hendrycks/math","n_code_links":5,"syntology":{"n_ran":3,"n_unverified":0,"n_samples":3,"n_pointer_only_licence":1}},{"rank_in_archive_order":124,"model":"GPT-2 (0.3B)","metrics":{"Accuracy":"6.2","Parameters (Billions)":"0.3"},"uses_additional_data":false,"paper_date":"2021-03-05","paper":"/paper/measuring-mathematical-problem-solving-with","paper_url":"https://arxiv.org/abs/2103.03874v2","paper_title":"Measuring Mathematical Problem Solving With the MATH Dataset","code":"https://github.com/hendrycks/math","n_code_links":5,"syntology":{"n_ran":3,"n_unverified":0,"n_samples":3,"n_pointer_only_licence":1}},{"rank_in_archive_order":125,"model":"GPT-3 13B","metrics":{"Accuracy":"5.6","Parameters (Billions)":"13"},"uses_additional_data":false,"paper_date":"2021-03-05","paper":"/paper/measuring-mathematical-problem-solving-with","paper_url":"https://arxiv.org/abs/2103.03874v2","paper_title":"Measuring Mathematical Problem Solving With the MATH Dataset","code":"https://github.com/hendrycks/math","n_code_links":5,"syntology":{"n_ran":3,"n_unverified":0,"n_samples":3,"n_pointer_only_licence":1}},{"rank_in_archive_order":126,"model":"PaLM 8B (fine-tuned)","metrics":{"Accuracy":"5.6","Parameters (Billions)":"8"},"uses_additional_data":false,"paper_date":"2022-06-29","paper":"/paper/solving-quantitative-reasoning-problems-with","paper_url":"https://arxiv.org/abs/2206.14858v2","paper_title":"Solving Quantitative Reasoning Problems with Language Models","code":"https://github.com/gair-nlp/abel","n_code_links":1,"syntology":null},{"rank_in_archive_order":127,"model":"GPT-2 (0.1B)","metrics":{"Accuracy":"5.4","Parameters (Billions)":"0.1"},"uses_additional_data":false,"paper_date":"2021-03-05","paper":"/paper/measuring-mathematical-problem-solving-with","paper_url":"https://arxiv.org/abs/2103.03874v2","paper_title":"Measuring Mathematical Problem Solving With the MATH Dataset","code":"https://github.com/hendrycks/math","n_code_links":5,"syntology":{"n_ran":3,"n_unverified":0,"n_samples":3,"n_pointer_only_licence":1}},{"rank_in_archive_order":128,"model":"GPT-3-175B (few-shot)","metrics":{"Accuracy":"5.2","Parameters (Billions)":"175"},"uses_additional_data":false,"paper_date":"2021-03-05","paper":"/paper/measuring-mathematical-problem-solving-with","paper_url":"https://arxiv.org/abs/2103.03874v2","paper_title":"Measuring Mathematical Problem Solving With the MATH Dataset","code":"https://github.com/hendrycks/math","n_code_links":5,"syntology":{"n_ran":3,"n_unverified":0,"n_samples":3,"n_pointer_only_licence":1}},{"rank_in_archive_order":129,"model":"GPT-3 175B (8-shot)","metrics":{"Accuracy":"5.2","Parameters (Billions)":"175"},"uses_additional_data":false,"paper_date":"2022-11-16","paper":"/paper/galactica-a-large-language-model-for-science-1","paper_url":"https://arxiv.org/abs/2211.09085v1","paper_title":"Galactica: A Large Language Model for Science","code":"https://github.com/paperswithcode/galai","n_code_links":1,"syntology":{"n_ran":0,"n_unverified":2,"n_samples":2,"n_pointer_only_licence":0}},{"rank_in_archive_order":130,"model":"PaLM 62B","metrics":{"Accuracy":"4.4","Parameters (Billions)":"62"},"uses_additional_data":false,"paper_date":"2022-06-29","paper":"/paper/solving-quantitative-reasoning-problems-with","paper_url":"https://arxiv.org/abs/2206.14858v2","paper_title":"Solving Quantitative Reasoning Problems with Language Models","code":"https://github.com/gair-nlp/abel","n_code_links":1,"syntology":null},{"rank_in_archive_order":131,"model":"LLaMA 13B","metrics":{"Accuracy":"3.9","Parameters (Billions)":"13"},"uses_additional_data":false,"paper_date":"2023-02-27","paper":"/paper/llama-open-and-efficient-foundation-language-1","paper_url":"https://arxiv.org/abs/2302.13971v1","paper_title":"LLaMA: Open and Efficient Foundation Language Models","code":"https://github.com/huggingface/transformers","n_code_links":57,"syntology":{"n_ran":26,"n_unverified":32,"n_samples":58,"n_pointer_only_licence":4}},{"rank_in_archive_order":132,"model":"GPT-3-13B (few-shot)","metrics":{"Accuracy":"3.0","Parameters (Billions)":"13"},"uses_additional_data":false,"paper_date":"2021-03-05","paper":"/paper/measuring-mathematical-problem-solving-with","paper_url":"https://arxiv.org/abs/2103.03874v2","paper_title":"Measuring Mathematical Problem Solving With the MATH Dataset","code":"https://github.com/hendrycks/math","n_code_links":5,"syntology":{"n_ran":3,"n_unverified":0,"n_samples":3,"n_pointer_only_licence":1}},{"rank_in_archive_order":133,"model":"LLaMA 7B","metrics":{"Accuracy":"2.9","Parameters (Billions)":"7"},"uses_additional_data":false,"paper_date":"2023-02-27","paper":"/paper/llama-open-and-efficient-foundation-language-1","paper_url":"https://arxiv.org/abs/2302.13971v1","paper_title":"LLaMA: Open and Efficient Foundation Language Models","code":"https://github.com/huggingface/transformers","n_code_links":57,"syntology":{"n_ran":26,"n_unverified":32,"n_samples":58,"n_pointer_only_licence":4}},{"rank_in_archive_order":134,"model":"GPT-3 2.7B","metrics":{"Accuracy":"2.9","Parameters (Billions)":"2.7"},"uses_additional_data":false,"paper_date":"2021-03-05","paper":"/paper/measuring-mathematical-problem-solving-with","paper_url":"https://arxiv.org/abs/2103.03874v2","paper_title":"Measuring Mathematical Problem Solving With the MATH Dataset","code":"https://github.com/hendrycks/math","n_code_links":5,"syntology":{"n_ran":3,"n_unverified":0,"n_samples":3,"n_pointer_only_licence":1}},{"rank_in_archive_order":135,"model":"PaLM 8B","metrics":{"Accuracy":"1.5","Parameters (Billions)":"8"},"uses_additional_data":false,"paper_date":"2022-06-29","paper":"/paper/solving-quantitative-reasoning-problems-with","paper_url":"https://arxiv.org/abs/2206.14858v2","paper_title":"Solving Quantitative Reasoning Problems with Language Models","code":"https://github.com/gair-nlp/abel","n_code_links":1,"syntology":null}],"since_archive":{"claim":"Results that newer papers report for their own method, placed here by Syntology. A model pointed at the cell in the paper's own table; the number was read from that cell and checked against this leaderboard's metric, dataset, split and scale; an independent check that saw this leaderboard's other rows and every other leaderboard on the same dataset accepted it. Not reviewed by the paper's authors or by the archive's editors, and not ranked against the archive rows.","extraction_file_present":true,"measurement":{"test_papers":883,"papers_with_output":881,"judged_true":108,"judged":110,"wilson95_lower":0.9361,"measured_on":"2026-09-24","frozen_commit":"0e3de0df94"},"measurement_note":"blind adjudication of accepted entries on a held-out split of archive papers, rules frozen before the test","coverage":{"sentence":"Syntology has checked 6,264 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","complete":false,"papers_newer_than_archive":9581,"papers_checked":6264,"papers_extracted_not_yet_verified":0,"boards_without_verdict":2,"papers_not_yet_extracted":3316},"order":"newest first by month (arXiv date, else the arXiv-id month), then arXiv id descending","columns":[],"entries":[]},"syntology":{"read_at":"2026-09-24T18:15:14+00:00","claim":"Per row: N of M harvested code samples from that row's paper executed on a synthesized fixture; the other M-N are unverified. Not a reproduction of the row's number; not a correctness claim. n_pointer_only_licence counts samples the site points at rather than redistributes (a licence axis, independent of ran/unverified).","rows_with_graph_line":76,"rows_with_any_sample_ran":61,"distinct_papers_with_graph_line":22,"distinct_papers_with_any_sample_ran":17,"samples_over_distinct_papers":{"n_ran":137,"n_unverified":115,"n_samples":252,"n_pointer_only_licence":78,"note":"each paper (arXiv id) counted once, however many rows it is behind; this is the page-level figure"},"samples_row_weighted":{"n_ran":560,"n_unverified":515,"n_samples":1075,"n_pointer_only_licence":200,"note":"row-weighted: a paper behind several rows is counted once per row; inflated relative to samples_over_distinct_papers by design, kept for readers summing the per-row syntology blocks"}}}