{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/autoformalize-mathematical-statements-by","title":"Autoformalize Mathematical Statements by Symbolic Equivalence and Semantic Consistency","arxiv_id":"2410.20936","date":"2024-10-28","proceeding":null,"authors":["Zenan Li","Yifan Wu","Zhaoyu Li","Xinming Wei","Xian Zhang","Fan Yang","Xiaoxing Ma"],"abstract":"Autoformalization, the task of automatically translating natural language descriptions into a formal language, poses a significant challenge across various domains, especially in mathematics. Recent advancements in large language models (LLMs) have unveiled their promising capabilities to formalize even competition-level math problems. However, we observe a considerable discrepancy between pass@1 and pass@k accuracies in LLM-generated formalizations. To address this gap, we introduce a novel framework that scores and selects the best result from k autoformalization candidates based on two complementary self-consistency methods: symbolic equivalence and semantic consistency. Elaborately, symbolic equivalence identifies the logical homogeneity among autoformalization candidates using automated theorem provers, and semantic consistency evaluates the preservation of the original meaning by informalizing the candidates and computing the similarity between the embeddings of the original and informalized texts. Our extensive experiments on the MATH and miniF2F datasets demonstrate that our approach significantly enhances autoformalization accuracy, achieving up to 0.22-1.35x relative improvements across various LLMs and baseline methods.","url_abs":"https://arxiv.org/abs/2410.20936v2","url_pdf":"https://arxiv.org/pdf/2410.20936v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"autoformalize-mathematical-statements-by","repo_url":"https://github.com/miracle-messi/isa-autoformal","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"math","task_name":"Math"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.20936","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.20936"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Miracle-Messi/Isa-AutoFormal","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/miracle-messi/isa-autoformal","reach":{"status":"ok"}}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"5c6cec005c819f34","entry":"custom_edit_distance","repo":"Miracle-Messi/Isa-AutoFormal","repo_kind":"official","path":"data-generation/bk.py","file_url":"https://github.com/Miracle-Messi/Isa-AutoFormal/blob/HEAD/data-generation/bk.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5c6cec005c819f34"}},{"code_sha256_prefix":"9ee896053d176b0c","entry":"get_json_files","repo":"Miracle-Messi/Isa-AutoFormal","repo_kind":"official","path":"equiv_checker/score_label.py","file_url":"https://github.com/Miracle-Messi/Isa-AutoFormal/blob/HEAD/equiv_checker/score_label.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9ee896053d176b0c"}},{"code_sha256_prefix":"2ded070fabe7bdc7","entry":"last_token_pool","repo":"Miracle-Messi/Isa-AutoFormal","repo_kind":"official","path":"equiv_checker/score_label.py","file_url":"https://github.com/Miracle-Messi/Isa-AutoFormal/blob/HEAD/equiv_checker/score_label.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2ded070fabe7bdc7"}},{"code_sha256_prefix":"353dee7130e71f3d","entry":"process_natural","repo":"Miracle-Messi/Isa-AutoFormal","repo_kind":"official","path":"data-generation/data_gen.py","file_url":"https://github.com/Miracle-Messi/Isa-AutoFormal/blob/HEAD/data-generation/data_gen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"353dee7130e71f3d"}},{"code_sha256_prefix":"b3be133415f3a632","entry":"simplify_statement","repo":"Miracle-Messi/Isa-AutoFormal","repo_kind":"official","path":"data-generation/bk.py","file_url":"https://github.com/Miracle-Messi/Isa-AutoFormal/blob/HEAD/data-generation/bk.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b3be133415f3a632"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}