{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2604-23002","title":"FormalScience: Scalable Human-in-the-Loop Autoformalisation of Science with Agentic Code Generation in Lean","arxiv_id":"2604.23002","date":"2026-04-24","proceeding":null,"authors":["Jordan Meadows","Lan Zhang","Andre Freitas"],"abstract":"Formalising informal mathematical reasoning into formally verifiable code is a significant challenge for large language models. In scientific fields such as physics, domain-specific machinery (\\textit{e.g.} Dirac notation, vector calculus) imposes additional formalisation challenges that modern LLMs and agentic approaches have yet to tackle. To aid autoformalisation in scientific domains, we present FormalScience; a domain-agnostic human-in-the-loop agentic pipeline that enables a single domain expert (without deep formal language experience) to produce \\textit{syntactically correct} and \\textit{semantically aligned} formal proofs of informal reasoning for low economic cost. Applying FormalScience to physics, we construct FormalPhysics, a dataset of 200 university-level (LaTeX) physics problems and solutions (primarily quantum mechanics and electromagnetism), along with their Lean4 formal representations. Compared to existing formal math benchmarks, FormalPhysics achieves perfect formal validity and exhibits greater statement complexity. We evaluate open-source models and proprietary systems on a statement autoformalisation task on our dataset via zero-shot prompting, self-refinement with error feedback, and a novel multi-stage agentic approach, and explore autoformalisation limitations in modern LLM-based approaches. We provide the first systematic characterisation of semantic drift in physics autoformalisation in terms of concepts such as notational collapse and abstraction elevation which reveals what formal language verifies when full semantic preservation is unattainable. We release the codebase together with an interactive UI-based FormalScience system which facilitates autoformalisation and theorem proving in scientific domains beyond physics.https://github.com/jmeadows17/formal-science","url_abs":"https://arxiv.org/abs/2604.23002","url_pdf":"https://arxiv.org/pdf/2604.23002","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2604.23002","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2604.23002"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/jmeadows17/formal-science","reach":null}],"summary":{"ran":10,"unverified":8},"by_repo_kind":{"found_in_text":{"samples":18,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":18,"samples":[{"code_sha256_prefix":"3f591af4077d1334","entry":"build_lean_prompt","repo":"jmeadows17/formal-science","repo_kind":"found_in_text","path":"src/app/lean_prompts.py","file_url":"https://github.com/jmeadows17/formal-science/blob/HEAD/src/app/lean_prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3f591af4077d1334"}},{"code_sha256_prefix":"292fbce17179b3f1","entry":"capture_msg","repo":"jmeadows17/formal-science","repo_kind":"found_in_text","path":"src/app/lean_app.py","file_url":"https://github.com/jmeadows17/formal-science/blob/HEAD/src/app/lean_app.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"292fbce17179b3f1"}},{"code_sha256_prefix":"3b923c6ca15115d6","entry":"flatten_structured_proofs","repo":"jmeadows17/formal-science","repo_kind":"found_in_text","path":"src/app/flatten_structured_proofs.py","file_url":"https://github.com/jmeadows17/formal-science/blob/HEAD/src/app/flatten_structured_proofs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3b923c6ca15115d6"}},{"code_sha256_prefix":"bea63caee1e73b2f","entry":"get_qa_batch","repo":"jmeadows17/formal-science","repo_kind":"found_in_text","path":"src/app/lean_prompts.py","file_url":"https://github.com/jmeadows17/formal-science/blob/HEAD/src/app/lean_prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bea63caee1e73b2f"}},{"code_sha256_prefix":"e4a71e1814a2f956","entry":"load_structured_proofs","repo":"jmeadows17/formal-science","repo_kind":"found_in_text","path":"src/app/postprocessing_pipeline.py","file_url":"https://github.com/jmeadows17/formal-science/blob/HEAD/src/app/postprocessing_pipeline.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e4a71e1814a2f956"}},{"code_sha256_prefix":"ff7e50034dfffbdd","entry":"postprocess_raw_dataset","repo":"jmeadows17/formal-science","repo_kind":"found_in_text","path":"src/qa/qa_postprocessing.py","file_url":"https://github.com/jmeadows17/formal-science/blob/HEAD/src/qa/qa_postprocessing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ff7e50034dfffbdd"}},{"code_sha256_prefix":"d57caf45c8ba6356","entry":"resolve_prompt","repo":"jmeadows17/formal-science","repo_kind":"found_in_text","path":"src/app/claude_generate_compile_loop.py","file_url":"https://github.com/jmeadows17/formal-science/blob/HEAD/src/app/claude_generate_compile_loop.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d57caf45c8ba6356"}},{"code_sha256_prefix":"6899f148b5ea5f1f","entry":"split_formal_proofs","repo":"jmeadows17/formal-science","repo_kind":"found_in_text","path":"src/app/flatten_structured_proofs.py","file_url":"https://github.com/jmeadows17/formal-science/blob/HEAD/src/app/flatten_structured_proofs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6899f148b5ea5f1f"}},{"code_sha256_prefix":"b854f3824585eab7","entry":"split_qas_markdown","repo":"jmeadows17/formal-science","repo_kind":"found_in_text","path":"src/qa/qa_postprocessing.py","file_url":"https://github.com/jmeadows17/formal-science/blob/HEAD/src/qa/qa_postprocessing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b854f3824585eab7"}},{"code_sha256_prefix":"05bc7565ec711eb5","entry":"write_flattened_structured_proofs","repo":"jmeadows17/formal-science","repo_kind":"found_in_text","path":"src/app/flatten_structured_proofs.py","file_url":"https://github.com/jmeadows17/formal-science/blob/HEAD/src/app/flatten_structured_proofs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"05bc7565ec711eb5"}},{"code_sha256_prefix":"ea8b2d7fb3799925","entry":"compile_lean","repo":"jmeadows17/formal-science","repo_kind":"found_in_text","path":"legacy/lean_utils.py","file_url":"https://github.com/jmeadows17/formal-science/blob/HEAD/legacy/lean_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ea8b2d7fb3799925"}},{"code_sha256_prefix":"f7546f4078fef874","entry":"compile_lean","repo":"jmeadows17/formal-science","repo_kind":"found_in_text","path":"src/app/compile_lean.py","file_url":"https://github.com/jmeadows17/formal-science/blob/HEAD/src/app/compile_lean.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f7546f4078fef874"}},{"code_sha256_prefix":"fd80353252a42ea7","entry":"default_few_shot_prompt_generation","repo":"jmeadows17/formal-science","repo_kind":"found_in_text","path":"src/qa/qa_prompt_generation.py","file_url":"https://github.com/jmeadows17/formal-science/blob/HEAD/src/qa/qa_prompt_generation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fd80353252a42ea7"}},{"code_sha256_prefix":"335d1ead6e46a569","entry":"load_formal_qa_data","repo":"jmeadows17/formal-science","repo_kind":"found_in_text","path":"src/app/postprocessing_pipeline.py","file_url":"https://github.com/jmeadows17/formal-science/blob/HEAD/src/app/postprocessing_pipeline.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"335d1ead6e46a569"}},{"code_sha256_prefix":"28ccff155d266348","entry":"load_postprocessing_audit","repo":"jmeadows17/formal-science","repo_kind":"found_in_text","path":"src/app/postprocessing_pipeline.py","file_url":"https://github.com/jmeadows17/formal-science/blob/HEAD/src/app/postprocessing_pipeline.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"28ccff155d266348"}},{"code_sha256_prefix":"a2df4f81e844be6b","entry":"load_qa_batches","repo":"jmeadows17/formal-science","repo_kind":"found_in_text","path":"src/app/lean_prompts.py","file_url":"https://github.com/jmeadows17/formal-science/blob/HEAD/src/app/lean_prompts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a2df4f81e844be6b"}},{"code_sha256_prefix":"e575724a8eaebe1d","entry":"postprocess_file","repo":"jmeadows17/formal-science","repo_kind":"found_in_text","path":"src/qa/qa_postprocessing.py","file_url":"https://github.com/jmeadows17/formal-science/blob/HEAD/src/qa/qa_postprocessing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e575724a8eaebe1d"}},{"code_sha256_prefix":"e47cd584f0ce0b55","entry":"run_loop","repo":"jmeadows17/formal-science","repo_kind":"found_in_text","path":"src/app/claude_generate_compile_loop.py","file_url":"https://github.com/jmeadows17/formal-science/blob/HEAD/src/app/claude_generate_compile_loop.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e47cd584f0ce0b55"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}