{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2511-18760","title":"HERMES: Towards Efficient and Verifiable Mathematical Reasoning in LLMs","arxiv_id":"2511.18760","date":"2025-11-24","proceeding":"ICML","authors":["Azim Ospanov","Zijin Feng","Jiacheng Sun","Haoli Bai","Xin Shen","Farzan Farnia"],"abstract":"Informal mathematics has been central to modern large language model (LLM) reasoning, offering flexibility and efficient construction of arguments. However, purely informal reasoning is prone to logical gaps and subtle errors that are difficult to detect and correct. In contrast, formal theorem proving provides rigorous, verifiable mathematical reasoning, where each inference step is checked by a trusted compiler, but lacks the exploratory freedom of informal problem-solving. This mismatch leaves current LLM-based math agents without a principled way to combine the strengths of both paradigms. In this work, we introduce Hermes, the first tool-assisted agent that explicitly interleaves informal reasoning with formally verified proofs in Lean. The framework performs intermediate formal checking to prevent reasoning drift and a memory module for proof continuity across multi-step reasoning chains, enabling both exploration and verification. We evaluate Hermes on four challenging mathematical reasoning benchmarks using LLMs of varying parameter scales, from small models to state-of-the-art systems. Across all settings, Hermes reliably improves the reasoning accuracy of base models while substantially reducing reasoning token usage and computational cost compared to reward-based approaches. On difficult datasets such as AIME and HARDMath2, Hermes@1 achieves up to a 40% accuracy improvement while using 80% fewer total inference FLOPs. When scaled at test time, Hermes@5 boosts accuracy further by 20%. The implementation and codebase are publicly available at https://github.com/aziksh-ospanov/HERMES.","url_abs":"https://arxiv.org/abs/2511.18760","url_pdf":"https://arxiv.org/pdf/2511.18760","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2511.18760","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2511.18760"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/aziksh-ospanov/HERMES","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":10},"by_repo_kind":{"found_in_text":{"samples":10,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"47f1f1e0c743c78f","entry":"call_scheduler","repo":"aziksh-ospanov/HERMES","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/aziksh-ospanov/HERMES/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"47f1f1e0c743c78f"}},{"code_sha256_prefix":"2759e9544307b4fb","entry":"extract_local_model_output","repo":"aziksh-ospanov/HERMES","repo_kind":"found_in_text","path":"utils/models.py","file_url":"https://github.com/aziksh-ospanov/HERMES/blob/HEAD/utils/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2759e9544307b4fb"}},{"code_sha256_prefix":"a53e6fa33bc0f732","entry":"is_deepseek_endpoint","repo":"aziksh-ospanov/HERMES","repo_kind":"found_in_text","path":"models/llm_client.py","file_url":"https://github.com/aziksh-ospanov/HERMES/blob/HEAD/models/llm_client.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a53e6fa33bc0f732"}},{"code_sha256_prefix":"8bb58cb2cdfa9f7d","entry":"llm_settings_from_config","repo":"aziksh-ospanov/HERMES","repo_kind":"found_in_text","path":"models/auxiliary_llm.py","file_url":"https://github.com/aziksh-ospanov/HERMES/blob/HEAD/models/auxiliary_llm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8bb58cb2cdfa9f7d"}},{"code_sha256_prefix":"f81a71bd89187804","entry":"looks_like_full_url","repo":"aziksh-ospanov/HERMES","repo_kind":"found_in_text","path":"models/llm_client.py","file_url":"https://github.com/aziksh-ospanov/HERMES/blob/HEAD/models/llm_client.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f81a71bd89187804"}},{"code_sha256_prefix":"bb74e5f2f4b39ba4","entry":"md_to_html","repo":"aziksh-ospanov/HERMES","repo_kind":"found_in_text","path":"render_readme_pdf.py","file_url":"https://github.com/aziksh-ospanov/HERMES/blob/HEAD/render_readme_pdf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bb74e5f2f4b39ba4"}},{"code_sha256_prefix":"1a185fd5ba9d831e","entry":"normalize_base_url","repo":"aziksh-ospanov/HERMES","repo_kind":"found_in_text","path":"models/llm_client.py","file_url":"https://github.com/aziksh-ospanov/HERMES/blob/HEAD/models/llm_client.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1a185fd5ba9d831e"}},{"code_sha256_prefix":"7c727b0f05d1a52a","entry":"timeout_handler","repo":"aziksh-ospanov/HERMES","repo_kind":"found_in_text","path":"multithread_support.py","file_url":"https://github.com/aziksh-ospanov/HERMES/blob/HEAD/multithread_support.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7c727b0f05d1a52a"}},{"code_sha256_prefix":"deaa4d18cf20a245","entry":"timeout_handler","repo":"aziksh-ospanov/HERMES","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/aziksh-ospanov/HERMES/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"deaa4d18cf20a245"}},{"code_sha256_prefix":"4b110f7f2fb2a56d","entry":"trace","repo":"aziksh-ospanov/HERMES","repo_kind":"found_in_text","path":"multithread_support.py","file_url":"https://github.com/aziksh-ospanov/HERMES/blob/HEAD/multithread_support.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4b110f7f2fb2a56d"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.AI","source":"arxiv_api"},"syntology_extracted_results":null}