{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/combibench-benchmarking-llm-capability-for","title":"CombiBench: Benchmarking LLM Capability for Combinatorial Mathematics","arxiv_id":"2505.03171","date":"2025-05-06","proceeding":null,"authors":["Junqi Liu","Xiaohan Lin","Jonas Bayer","Yael Dillies","Weijie Jiang","Xiaodan Liang","Roman Soletskyi","Haiming Wang","Yunzhou Xie","Beibei Xiong","Zhengfeng Yang","Jujian Zhang","Lihong Zhi","Jia Li","Zhengying Liu"],"abstract":"Neurosymbolic approaches integrating large language models with formal reasoning have recently achieved human-level performance on mathematics competition problems in algebra, geometry and number theory. In comparison, combinatorics remains a challenging domain, characterized by a lack of appropriate benchmarks and theorem libraries. To address this gap, we introduce CombiBench, a comprehensive benchmark comprising 100 combinatorial problems, each formalized in Lean~4 and paired with its corresponding informal statement. The problem set covers a wide spectrum of difficulty levels, ranging from middle school to IMO and university level, and span over ten combinatorial topics. CombiBench is suitable for testing IMO solving capabilities since it includes all IMO combinatorial problems since 2000 (except IMO 2004 P3 as its statement contain an images). Furthermore, we provide a comprehensive and standardized evaluation framework, dubbed Fine-Eval (for $\\textbf{F}$ill-in-the-blank $\\textbf{in}$ L$\\textbf{e}$an Evaluation), for formal mathematics. It accommodates not only proof-based problems but also, for the first time, the evaluation of fill-in-the-blank questions. Using Fine-Eval as the evaluation method and Kimina Lean Server as the backend, we benchmark several LLMs on CombiBench and observe that their capabilities for formally solving combinatorial problems remain limited. Among all models tested (none of which has been trained for this particular task), Kimina-Prover attains the best results, solving 7 problems (out of 100) under both ``with solution'' and ``without solution'' scenarios. We open source the benchmark dataset alongside with the code of the proposed evaluation method at https://github.com/MoonshotAI/CombiBench/.","url_abs":"https://arxiv.org/abs/2505.03171v1","url_pdf":"https://arxiv.org/pdf/2505.03171v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"combibench-benchmarking-llm-capability-for","repo_url":"https://github.com/moonshotai/combibench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2505.03171","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.03171"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/moonshotai/combibench","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"214354418ec1f3ef","entry":"add_sorry_to_statement","repo":"moonshotai/combibench","repo_kind":"official","path":"evaluation/util.py","file_url":"https://github.com/moonshotai/combibench/blob/HEAD/evaluation/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"214354418ec1f3ef"}},{"code_sha256_prefix":"cad080a5be6a1b23","entry":"batch_verify_proof","repo":"moonshotai/combibench","repo_kind":"official","path":"evaluation/client/lean_client.py","file_url":"https://github.com/moonshotai/combibench/blob/HEAD/evaluation/client/lean_client.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cad080a5be6a1b23"}},{"code_sha256_prefix":"8587e1e7b2168fd5","entry":"process_batch","repo":"moonshotai/combibench","repo_kind":"official","path":"evaluation/client/lean_client.py","file_url":"https://github.com/moonshotai/combibench/blob/HEAD/evaluation/client/lean_client.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8587e1e7b2168fd5"}},{"code_sha256_prefix":"7b8ef494fa71dcfd","entry":"process_batches","repo":"moonshotai/combibench","repo_kind":"official","path":"evaluation/client/lean_client.py","file_url":"https://github.com/moonshotai/combibench/blob/HEAD/evaluation/client/lean_client.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b8ef494fa71dcfd"}},{"code_sha256_prefix":"8f8f8f9de8f5a5b4","entry":"remove_comments","repo":"moonshotai/combibench","repo_kind":"official","path":"evaluation/util.py","file_url":"https://github.com/moonshotai/combibench/blob/HEAD/evaluation/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8f8f8f9de8f5a5b4"}},{"code_sha256_prefix":"20cb3b242357f7cd","entry":"remove_sorry_statements","repo":"moonshotai/combibench","repo_kind":"official","path":"evaluation/util.py","file_url":"https://github.com/moonshotai/combibench/blob/HEAD/evaluation/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"20cb3b242357f7cd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}