{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lyra-orchestrating-dual-correction-in","title":"Lyra: Orchestrating Dual Correction in Automated Theorem Proving","arxiv_id":"2309.15806","date":"2023-09-27","proceeding":null,"authors":["Chuanyang Zheng","Haiming Wang","Enze Xie","Zhengying Liu","Jiankai Sun","Huajian Xin","Jianhao Shen","Zhenguo Li","Yu Li"],"abstract":"Large Language Models (LLMs) present an intriguing avenue for exploration in the field of formal theorem proving. Nevertheless, their full potential, particularly concerning the mitigation of hallucinations and refinement through prover error messages, remains an area that has yet to be thoroughly investigated. To enhance the effectiveness of LLMs in the field, we introduce the Lyra, a new framework that employs two distinct correction mechanisms: Tool Correction (TC) and Conjecture Correction (CC). To implement Tool Correction in the post-processing of formal proofs, we leverage prior knowledge to utilize predefined prover tools (e.g., Sledgehammer) for guiding the replacement of incorrect tools. Tool Correction significantly contributes to mitigating hallucinations, thereby improving the overall accuracy of the proof. In addition, we introduce Conjecture Correction, an error feedback mechanism designed to interact with prover to refine formal proof conjectures with prover error messages. Compared to the previous refinement framework, the proposed Conjecture Correction refines generation with instruction but does not collect paired (generation, error & refinement) prompts. Our method has achieved state-of-the-art (SOTA) performance on both miniF2F validation (48.0% -> 55.3%) and test (45.5% -> 51.2%). We also present 3 IMO problems solved by Lyra. We believe Tool Correction (post-process for hallucination mitigation) and Conjecture Correction (subgoal adjustment from interaction with environment) could provide a promising avenue for future research in this field.","url_abs":"https://arxiv.org/abs/2309.15806v4","url_pdf":"https://arxiv.org/pdf/2309.15806v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lyra-orchestrating-dual-correction-in","repo_url":"https://github.com/chuanyang-zheng/lyra-theorem-prover","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"automated-theorem-proving","task_name":"Automated Theorem Proving"},{"task_slug":"hallucination","task_name":"Hallucination"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/automated-theorem-proving-on-minif2f-test","task":"Automated Theorem Proving","dataset":"miniF2F-test","model":"Lyra + GPT-4","rank_in_archive_order":6,"of":29,"metrics":{"ITP":"Isabelle","Pass@100":"47.1","cumulative":"47.1"},"uses_additional_data":false},{"leaderboard":"/sota/automated-theorem-proving-on-minif2f-valid","task":"Automated Theorem Proving","dataset":"miniF2F-valid","model":"Lyra + GPT-4","rank_in_archive_order":9,"of":10,"metrics":{"Pass@100":"52.0"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2309.15806","atlas_url":"https://app.syntology.ai/?focus=2309.15806","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.15806"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chuanyang-zheng/lyra-theorem-prover","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":7,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"15fd5c7c3c95ffe6","entry":"MATH_example_template","repo":"chuanyang-zheng/lyra-theorem-prover","repo_kind":"official","path":"src/autoformalization/prompts.py","file_url":"https://github.com/chuanyang-zheng/lyra-theorem-prover/blob/HEAD/src/autoformalization/prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"15fd5c7c3c95ffe6"}},{"code_sha256_prefix":"bd6230cf4c7dd610","entry":"apply_subst","repo":"chuanyang-zheng/lyra-theorem-prover","repo_kind":"official","path":"formal_system_servers/mm_server_backend.py","file_url":"https://github.com/chuanyang-zheng/lyra-theorem-prover/blob/HEAD/formal_system_servers/mm_server_backend.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bd6230cf4c7dd610"}},{"code_sha256_prefix":"3a7d6557aa829cee","entry":"get_finish","repo":"chuanyang-zheng/lyra-theorem-prover","repo_kind":"official","path":"src/autoformalization/utils.py","file_url":"https://github.com/chuanyang-zheng/lyra-theorem-prover/blob/HEAD/src/autoformalization/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3a7d6557aa829cee"}},{"code_sha256_prefix":"e10458c4731d6819","entry":"get_problem_names","repo":"chuanyang-zheng/lyra-theorem-prover","repo_kind":"official","path":"src/autoformalization/utils.py","file_url":"https://github.com/chuanyang-zheng/lyra-theorem-prover/blob/HEAD/src/autoformalization/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e10458c4731d6819"}},{"code_sha256_prefix":"d1ea6db0edebefb1","entry":"parse_tactic","repo":"chuanyang-zheng/lyra-theorem-prover","repo_kind":"official","path":"formal_system_servers/mm_server_backend.py","file_url":"https://github.com/chuanyang-zheng/lyra-theorem-prover/blob/HEAD/formal_system_servers/mm_server_backend.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d1ea6db0edebefb1"}},{"code_sha256_prefix":"0c6d2acafc44ef56","entry":"prepare_generated_proof","repo":"chuanyang-zheng/lyra-theorem-prover","repo_kind":"official","path":"run_interact.py","file_url":"https://github.com/chuanyang-zheng/lyra-theorem-prover/blob/HEAD/run_interact.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0c6d2acafc44ef56"}},{"code_sha256_prefix":"39367ca7096740ac","entry":"recursive_extract","repo":"chuanyang-zheng/lyra-theorem-prover","repo_kind":"official","path":"src/autoformalization/utils.py","file_url":"https://github.com/chuanyang-zheng/lyra-theorem-prover/blob/HEAD/src/autoformalization/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"39367ca7096740ac"}},{"code_sha256_prefix":"6ef513fef14838a6","entry":"parse_tactic_state","repo":"chuanyang-zheng/lyra-theorem-prover","repo_kind":"official","path":"formal_system_servers/mm_server_backend.py","file_url":"https://github.com/chuanyang-zheng/lyra-theorem-prover/blob/HEAD/formal_system_servers/mm_server_backend.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6ef513fef14838a6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}