{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2603-02668","title":"SorryDB: Can AI Provers Complete Real-World Lean Theorems?","arxiv_id":"2603.02668","date":"2026-03-03","proceeding":"ICML","authors":["Austin Letson","Leopoldo Sarra","Auguste Poiroux","Oliver Dressler","Paul Lezeau","Dhyan Aranha","Frederick Pu","Aaron Hill","Miguel Corredera Hidalgo","Julian Berman","George Tsoukalas","Lenny Taelman"],"abstract":"We present SorryDB, a dynamically-updating benchmark of open Lean tasks drawn from 78 real world formalization projects on GitHub. Unlike existing static benchmarks, often composed of competition problems, hillclimbing the SorryDB benchmark will yield tools that are aligned to the community needs, more usable by mathematicians, and more capable of understanding complex dependencies. Moreover, by providing a continuously updated stream of tasks, SorryDB mitigates test-set contamination and offers a robust metric for an agent's ability to contribute to novel formal mathematics projects. We evaluate a collection of approaches, including generalist large language models, agentic approaches, and specialized symbolic provers, over a selected snapshot of 1000 tasks from SorryDB. We show that current approaches are complementary: even though an agentic approach based on Gemini Flash is the most performant, it is not strictly better than other off-the-shelf large-language models, specialized provers, or even a curated list of Lean tactics.","url_abs":"https://arxiv.org/abs/2603.02668","url_pdf":"https://arxiv.org/pdf/2603.02668","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2603.02668","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2603.02668"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/SorryDB/SorryDB","reach":null}],"summary":{"unverified":3},"by_repo_kind":{"found_in_text":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7f4212ad0d5a1bf8","entry":"clone_or_fetch_repo","repo":"SorryDB/SorryDB","repo_kind":"found_in_text","path":"sorrydb/cli/analyze_reservoir_history.py","file_url":"https://github.com/SorryDB/SorryDB/blob/HEAD/sorrydb/cli/analyze_reservoir_history.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7f4212ad0d5a1bf8"}},{"code_sha256_prefix":"247b37a16c439480","entry":"count_sorries_in_line","repo":"SorryDB/SorryDB","repo_kind":"found_in_text","path":"sorrydb/cli/analyze_reservoir_history.py","file_url":"https://github.com/SorryDB/SorryDB/blob/HEAD/sorrydb/cli/analyze_reservoir_history.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"247b37a16c439480"}},{"code_sha256_prefix":"cc3effc8dd1cd19d","entry":"get_repo_name_from_url","repo":"SorryDB/SorryDB","repo_kind":"found_in_text","path":"sorrydb/cli/analyze_reservoir_history.py","file_url":"https://github.com/SorryDB/SorryDB/blob/HEAD/sorrydb/cli/analyze_reservoir_history.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cc3effc8dd1cd19d"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}