{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/minif2f-a-cross-system-benchmark-for-formal","title":"MiniF2F: a cross-system benchmark for formal Olympiad-level mathematics","arxiv_id":"2109.00110","date":"2021-08-31","proceeding":"ICLR 2022 4","authors":["Kunhao Zheng","Jesse Michael Han","Stanislas Polu"],"abstract":"We present miniF2F, a dataset of formal Olympiad-level mathematics problems statements intended to provide a unified cross-system benchmark for neural theorem proving. The miniF2F benchmark currently targets Metamath, Lean, Isabelle (partially) and HOL Light (partially) and consists of 488 problem statements drawn from the AIME, AMC, and the International Mathematical Olympiad (IMO), as well as material from high-school and undergraduate mathematics courses. We report baseline results using GPT-f, a neural theorem prover based on GPT-3 and provide an analysis of its performance. We intend for miniF2F to be a community-driven effort and hope that our benchmark will help spur advances in neural theorem proving.","url_abs":"https://arxiv.org/abs/2109.00110v2","url_pdf":"https://arxiv.org/pdf/2109.00110v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"minif2f-a-cross-system-benchmark-for-formal","repo_url":"https://github.com/openai/minif2f","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"minif2f-a-cross-system-benchmark-for-formal","repo_url":"https://github.com/facebookresearch/minif2f","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"minif2f-a-cross-system-benchmark-for-formal","repo_url":"https://github.com/rah4927/lean-dojo-mew","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"minif2f-a-cross-system-benchmark-for-formal","repo_url":"https://github.com/yangky11/minif2f-lean4","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"automated-theorem-proving","task_name":"Automated Theorem Proving"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[{"slug":"minif2f","name":"MiniF2F","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/automated-theorem-proving-on-minif2f-test","task":"Automated Theorem Proving","dataset":"miniF2F-test","model":"Lean GPT-f","rank_in_archive_order":18,"of":29,"metrics":{"ITP":"Lean","Pass@1":"24.6","Pass@32":"29.2","cumulative":"29.2"},"uses_additional_data":false},{"leaderboard":"/sota/automated-theorem-proving-on-minif2f-test","task":"Automated Theorem Proving","dataset":"miniF2F-test","model":"Lean tidy","rank_in_archive_order":26,"of":29,"metrics":{"ITP":"Lean","Pass@1":"18","cumulative":"18"},"uses_additional_data":false},{"leaderboard":"/sota/automated-theorem-proving-on-minif2f-test","task":"Automated Theorem Proving","dataset":"miniF2F-test","model":"Metamath GPT-f","rank_in_archive_order":29,"of":29,"metrics":{"ITP":"Metamath","Pass@1":"1.3","cumulative":"1.6"},"uses_additional_data":false},{"leaderboard":"/sota/automated-theorem-proving-on-minif2f-valid","task":"Automated Theorem Proving","dataset":"miniF2F-valid","model":"Lean GPT-f","rank_in_archive_order":1,"of":10,"metrics":{"Pass@1":"23.9","Pass@8":"29.3"},"uses_additional_data":true},{"leaderboard":"/sota/automated-theorem-proving-on-minif2f-valid","task":"Automated Theorem Proving","dataset":"miniF2F-valid","model":"Metamath GPT-f","rank_in_archive_order":2,"of":10,"metrics":{"Pass@1":"1","Pass@8":"2"},"uses_additional_data":false},{"leaderboard":"/sota/automated-theorem-proving-on-minif2f-valid","task":"Automated Theorem Proving","dataset":"miniF2F-valid","model":"Lean tidy","rank_in_archive_order":7,"of":10,"metrics":{"Pass@1":"16.8"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2109.00110","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2109.00110"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rah4927/lean-dojo-mew","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/minif2f","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/openai/minif2f","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yangky11/minif2f-lean4","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":3},"by_repo_kind":{"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6cd16db0c1444e10","entry":"reserved_notation_check","repo":"facebookresearch/minif2f","repo_kind":"listed","path":"lean/scripts/lint_style.py","file_url":"https://github.com/facebookresearch/minif2f/blob/HEAD/lean/scripts/lint_style.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6cd16db0c1444e10"}},{"code_sha256_prefix":"26a8582dd6f1cac7","entry":"set_option_check","repo":"facebookresearch/minif2f","repo_kind":"listed","path":"lean/scripts/lint_style.py","file_url":"https://github.com/facebookresearch/minif2f/blob/HEAD/lean/scripts/lint_style.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"26a8582dd6f1cac7"}},{"code_sha256_prefix":"9a6fcf24bb8c49fb","entry":"small_alpha_vrachy_check","repo":"facebookresearch/minif2f","repo_kind":"listed","path":"lean/scripts/lint_style.py","file_url":"https://github.com/facebookresearch/minif2f/blob/HEAD/lean/scripts/lint_style.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a6fcf24bb8c49fb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}