{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/minictx-neural-theorem-proving-with-long","title":"miniCTX: Neural Theorem Proving with (Long-)Contexts","arxiv_id":"2408.03350","date":"2024-08-05","proceeding":null,"authors":["Jiewen Hu","Thomas Zhu","Sean Welleck"],"abstract":"Real-world formal theorem proving often depends on a wealth of context, including definitions, lemmas, comments, file structure, and other information. We introduce miniCTX, which tests a model's ability to prove formal mathematical theorems that depend on new context that is not seen during training. miniCTX contains theorems sourced from real Lean projects and textbooks, each associated with a context that can span tens of thousands of tokens. Models are tasked with proving a theorem given access to code from the theorem's repository, which contains context that is needed for the proof. As a baseline for miniCTX, we tested fine-tuning and prompting methods that condition theorem proving on preceding context. Both approaches substantially outperform traditional methods that rely solely on state information. We found that this ability to use context is not captured by previous benchmarks such as miniF2F. Alongside miniCTX, we offer ntp-toolkit for automatically extracting and annotating theorem proving data, making it easy to add new projects into miniCTX to ensure that contexts are not seen during training. miniCTX offers a challenging and realistic evaluation of neural theorem provers.","url_abs":"https://arxiv.org/abs/2408.03350v2","url_pdf":"https://arxiv.org/pdf/2408.03350v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"minictx-neural-theorem-proving-with-long","repo_url":"https://github.com/cmu-l3/minictx-eval","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"minictx-neural-theorem-proving-with-long","repo_url":"https://github.com/cmu-l3/ntp-toolkit","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"minictx-neural-theorem-proving-with-long","repo_url":"https://github.com/HEPLean/HepLean","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"minictx-neural-theorem-proving-with-long","repo_url":"https://github.com/heplean/physlean","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"automated-theorem-proving","task_name":"Automated Theorem Proving"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2408.03350","atlas_url":"https://app.syntology.ai/?focus=2408.03350","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.03350"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cmu-l3/minictx-eval","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/HEPLean/HepLean","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cmu-l3/ntp-toolkit","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/heplean/physlean","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":5},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"61b52f161689f7c9","entry":"discard_after_marker","repo":"cmu-l3/minictx-eval","repo_kind":"official","path":"check.py","file_url":"https://github.com/cmu-l3/minictx-eval/blob/HEAD/check.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"61b52f161689f7c9"}},{"code_sha256_prefix":"61b6e1d9be3efdf1","entry":"encode","repo":"cmu-l3/minictx-eval","repo_kind":"official","path":"premise_retriever.py","file_url":"https://github.com/cmu-l3/minictx-eval/blob/HEAD/premise_retriever.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"61b6e1d9be3efdf1"}},{"code_sha256_prefix":"9a7bd3f70d3b094a","entry":"get_premises","repo":"cmu-l3/minictx-eval","repo_kind":"official","path":"check.py","file_url":"https://github.com/cmu-l3/minictx-eval/blob/HEAD/check.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9a7bd3f70d3b094a"}},{"code_sha256_prefix":"5a9ad2c905186a2f","entry":"retrieve","repo":"cmu-l3/minictx-eval","repo_kind":"official","path":"premise_retriever.py","file_url":"https://github.com/cmu-l3/minictx-eval/blob/HEAD/premise_retriever.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5a9ad2c905186a2f"}},{"code_sha256_prefix":"39dea6aeb3c26e50","entry":"truncate_middle","repo":"cmu-l3/minictx-eval","repo_kind":"official","path":"check.py","file_url":"https://github.com/cmu-l3/minictx-eval/blob/HEAD/check.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39dea6aeb3c26e50"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}