{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/measuring-mathematical-problem-solving-with","title":"Measuring Mathematical Problem Solving With the MATH Dataset","arxiv_id":"2103.03874","date":"2021-03-05","proceeding":null,"authors":["Dan Hendrycks","Collin Burns","Saurav Kadavath","Akul Arora","Steven Basart","Eric Tang","Dawn Song","Jacob Steinhardt"],"abstract":"Many intellectual endeavors require mathematical problem solving, but this skill remains beyond the capabilities of computers. To measure this ability in machine learning models, we introduce MATH, a new dataset of 12,500 challenging competition mathematics problems. Each problem in MATH has a full step-by-step solution which can be used to teach models to generate answer derivations and explanations. To facilitate future research and increase accuracy on MATH, we also contribute a large auxiliary pretraining dataset which helps teach models the fundamentals of mathematics. Even though we are able to increase accuracy on MATH, our results show that accuracy remains relatively low, even with enormous Transformer models. Moreover, we find that simply increasing budgets and model parameter counts will be impractical for achieving strong mathematical reasoning if scaling trends continue. While scaling Transformers is automatically solving most other text-based tasks, scaling is not currently solving MATH. To have more traction on mathematical problem solving we will likely need new algorithmic advancements from the broader research community.","url_abs":"https://arxiv.org/abs/2103.03874v2","url_pdf":"https://arxiv.org/pdf/2103.03874v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"measuring-mathematical-problem-solving-with","repo_url":"https://github.com/hendrycks/math","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"measuring-mathematical-problem-solving-with","repo_url":"https://github.com/facebookresearch/minif2f","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"measuring-mathematical-problem-solving-with","repo_url":"https://github.com/openai/minif2f","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"measuring-mathematical-problem-solving-with","repo_url":"https://github.com/radzim/obfuscated","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"measuring-mathematical-problem-solving-with","repo_url":"https://github.com/rah4927/lean-dojo-mew","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"math","task_name":"Math"},{"task_slug":"math-word-problem-solving","task_name":"Math Word Problem Solving"},{"task_slug":"mathematical-problem-solving","task_name":"Mathematical Problem-Solving"},{"task_slug":"mathematical-reasoning","task_name":"Mathematical Reasoning"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[{"slug":"amps","name":"AMPS","full_name":"Auxiliary Mathematics Problems and Solutions"},{"slug":"math","name":"MATH","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/math-word-problem-solving-on-math","task":"Math Word Problem Solving","dataset":"MATH","model":"GPT-2 (1.5B)","rank_in_archive_order":122,"of":135,"metrics":{"Accuracy":"6.9","Parameters (Billions)":"1.5"},"uses_additional_data":false},{"leaderboard":"/sota/math-word-problem-solving-on-math","task":"Math Word Problem Solving","dataset":"MATH","model":"GPT-2 (0.7B)","rank_in_archive_order":123,"of":135,"metrics":{"Accuracy":"6.4","Parameters (Billions)":"0.7"},"uses_additional_data":false},{"leaderboard":"/sota/math-word-problem-solving-on-math","task":"Math Word Problem Solving","dataset":"MATH","model":"GPT-2 (0.3B)","rank_in_archive_order":124,"of":135,"metrics":{"Accuracy":"6.2","Parameters (Billions)":"0.3"},"uses_additional_data":false},{"leaderboard":"/sota/math-word-problem-solving-on-math","task":"Math Word Problem Solving","dataset":"MATH","model":"GPT-3 13B","rank_in_archive_order":125,"of":135,"metrics":{"Accuracy":"5.6","Parameters (Billions)":"13"},"uses_additional_data":false},{"leaderboard":"/sota/math-word-problem-solving-on-math","task":"Math Word Problem Solving","dataset":"MATH","model":"GPT-2 (0.1B)","rank_in_archive_order":127,"of":135,"metrics":{"Accuracy":"5.4","Parameters (Billions)":"0.1"},"uses_additional_data":false},{"leaderboard":"/sota/math-word-problem-solving-on-math","task":"Math Word Problem Solving","dataset":"MATH","model":"GPT-3-175B (few-shot)","rank_in_archive_order":128,"of":135,"metrics":{"Accuracy":"5.2","Parameters (Billions)":"175"},"uses_additional_data":false},{"leaderboard":"/sota/math-word-problem-solving-on-math","task":"Math Word Problem Solving","dataset":"MATH","model":"GPT-3-13B (few-shot)","rank_in_archive_order":132,"of":135,"metrics":{"Accuracy":"3.0","Parameters (Billions)":"13"},"uses_additional_data":false},{"leaderboard":"/sota/math-word-problem-solving-on-math","task":"Math Word Problem Solving","dataset":"MATH","model":"GPT-3 2.7B","rank_in_archive_order":134,"of":135,"metrics":{"Accuracy":"2.9","Parameters (Billions)":"2.7"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2103.03874","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2103.03874"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Khan/khan-exercises","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rah4927/lean-dojo-mew","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hendrycks/math","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/minif2f","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/radzim/obfuscated","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/openai/minif2f","reach":{"status":"ok"}}],"summary":{"ran_fixture":1,"ran_draft_wrong":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"48d671a5de2eef0a","entry":"dict_to_gpu","repo":"hendrycks/math","repo_kind":"official","path":"modeling/eval_math_gpt.py","file_url":"https://github.com/hendrycks/math/blob/HEAD/modeling/eval_math_gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"48d671a5de2eef0a"}},{"code_sha256_prefix":"203cd02db658aac9","entry":"get_level_type","repo":"hendrycks/math","repo_kind":"official","path":"modeling/eval_math_gpt.py","file_url":"https://github.com/hendrycks/math/blob/HEAD/modeling/eval_math_gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"203cd02db658aac9"}},{"code_sha256_prefix":"f3bbe264b05aadd3","entry":"remove_boxed","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"f3bbe264b05aadd3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}