{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2606-29493","title":"Faults in Our Formal Benchmarking: Dataset Defects and Evaluation Failures in Lean Theorem Proving","arxiv_id":"2606.29493","date":"2026-06-28","proceeding":"ICML","authors":["Pawan Sasanka Ammanamanchi","Siddharth Bhat","Stella Biderman"],"abstract":"Benchmarks for LLM-assisted theorem proving in Lean are often treated as intrinsically reliable because every solved instance comes with a machine-checked proof. However, the kernel only checks that a proof establishes a \\emph{formal} statement; it does not verify that the statement faithfully encodes the intended informal problem, nor that evaluation harnesses are robust to trivial or adversarial solutions. We audit five widely used Lean theorem-proving benchmarks and their forks, using corpus-scale static checkers to surface 4,833 findings, including 398 mechanically certified issues such as counterexamples, vacuous theorems, and unsound axioms. We also document semantic defects such as missing hypotheses, problem simplification, incomplete or incorrect translations, and Lean-specific specification hazards. Beyond dataset construction, we survey evaluation-time failure modes and show, on corrected subsets, that defects can both inflate and deflate reported prover scores. We propose a fault taxonomy, a suite of automated checkers and recall-oriented semantic audit prompts, and release standards to guide the creation of formal math datasets and to make evaluation more reproducible and trustworthy. Our checkers, audit prompts, and corrected dataset snapshots are available at https://github.com/Shashi456/atp-checkers.","url_abs":"https://arxiv.org/abs/2606.29493","url_pdf":"https://arxiv.org/pdf/2606.29493","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2606.29493","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2606.29493"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/leanprover/lean4","reach":null}],"summary":{"ran":8,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":10,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6665056f19db0e93","entry":"color","repo":"leanprover/lean4","repo_kind":"found_in_text","path":"script/build_artifact.py","file_url":"https://github.com/leanprover/lean4/blob/HEAD/script/build_artifact.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6665056f19db0e93"}},{"code_sha256_prefix":"b92fe2af864b084e","entry":"demangle_lean_name","repo":"leanprover/lean4","repo_kind":"found_in_text","path":"script/profiler/lean_demangle.py","file_url":"https://github.com/leanprover/lean4/blob/HEAD/script/profiler/lean_demangle.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b92fe2af864b084e"}},{"code_sha256_prefix":"b5875af47ee85e67","entry":"get_cache_path","repo":"leanprover/lean4","repo_kind":"found_in_text","path":"script/build_artifact.py","file_url":"https://github.com/leanprover/lean4/blob/HEAD/script/build_artifact.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b5875af47ee85e67"}},{"code_sha256_prefix":"687def3c02ea528c","entry":"main","repo":"leanprover/lean4","repo_kind":"found_in_text","path":"script/gen_constants_cpp.py","file_url":"https://github.com/leanprover/lean4/blob/HEAD/script/gen_constants_cpp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"687def3c02ea528c"}},{"code_sha256_prefix":"193ca90e21e7f2eb","entry":"main","repo":"leanprover/lean4","repo_kind":"found_in_text","path":"script/gen_tokens_cpp.py","file_url":"https://github.com/leanprover/lean4/blob/HEAD/script/gen_tokens_cpp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"193ca90e21e7f2eb"}},{"code_sha256_prefix":"818ae0ad863cbdd8","entry":"to_c_const","repo":"leanprover/lean4","repo_kind":"found_in_text","path":"script/gen_constants_cpp.py","file_url":"https://github.com/leanprover/lean4/blob/HEAD/script/gen_constants_cpp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"818ae0ad863cbdd8"}},{"code_sha256_prefix":"26eb017b44e66317","entry":"to_c_const","repo":"leanprover/lean4","repo_kind":"found_in_text","path":"script/gen_tokens_cpp.py","file_url":"https://github.com/leanprover/lean4/blob/HEAD/script/gen_tokens_cpp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"26eb017b44e66317"}},{"code_sha256_prefix":"8664291b9fcd8fb8","entry":"truncate","repo":"leanprover/lean4","repo_kind":"found_in_text","path":"script/junit_embed_output.py","file_url":"https://github.com/leanprover/lean4/blob/HEAD/script/junit_embed_output.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8664291b9fcd8fb8"}},{"code_sha256_prefix":"43a715be97173929","entry":"github_api_request","repo":"leanprover/lean4","repo_kind":"found_in_text","path":"script/build_artifact.py","file_url":"https://github.com/leanprover/lean4/blob/HEAD/script/build_artifact.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"43a715be97173929"}},{"code_sha256_prefix":"75564b0335312c63","entry":"parse","repo":"leanprover/lean4","repo_kind":"found_in_text","path":"script/diff_changelogs.py","file_url":"https://github.com/leanprover/lean4/blob/HEAD/script/diff_changelogs.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"75564b0335312c63"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.AI","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}