{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/heurigym-an-agentic-benchmark-for-llm-crafted","title":"HeuriGym: An Agentic Benchmark for LLM-Crafted Heuristics in Combinatorial Optimization","arxiv_id":"2506.07972","date":"2025-06-09","proceeding":null,"authors":["Hongzheng Chen","Yingheng Wang","Yaohui Cai","Hins Hu","Jiajie Li","Shirley Huang","Chenhui Deng","Rongjian Liang","Shufeng Kong","Haoxing Ren","Samitha Samaranayake","Carla P. Gomes","Zhiru Zhang"],"abstract":"While Large Language Models (LLMs) have demonstrated significant advancements in reasoning and agent-based problem-solving, current evaluation methodologies fail to adequately assess their capabilities: existing benchmarks either rely on closed-ended questions prone to saturation and memorization, or subjective comparisons that lack consistency and rigor. In this work, we introduce HeuriGym, an agentic framework designed for evaluating heuristic algorithms generated by LLMs for combinatorial optimization problems, characterized by clearly defined objectives and expansive solution spaces. HeuriGym empowers LLMs to propose heuristics, receive evaluative feedback via code execution, and iteratively refine their solutions. We evaluate nine state-of-the-art models on nine problems across domains such as computer systems, logistics, and biology, exposing persistent limitations in tool use, planning, and adaptive reasoning. To quantify performance, we propose the Quality-Yield Index (QYI), a metric that captures both solution pass rate and quality. Even top models like GPT-o4-mini-high and Gemini-2.5-Pro attain QYI scores of only 0.6, well below the expert baseline of 1. Our open-source benchmark aims to guide the development of LLMs toward more effective and realistic problem-solving in scientific and engineering domains.","url_abs":"https://arxiv.org/abs/2506.07972v1","url_pdf":"https://arxiv.org/pdf/2506.07972v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"heurigym-an-agentic-benchmark-for-llm-crafted","repo_url":"https://github.com/cornell-zhang/heurigym","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"combinatorial-optimization","task_name":"Combinatorial Optimization"},{"task_slug":"memorization","task_name":"Memorization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.07972","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.07972"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cornell-zhang/heurigym","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":12},"by_repo_kind":{"official":{"samples":12,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6fcd80e8413e9b07","entry":"calculate_cost","repo":"cornell-zhang/heurigym","repo_kind":"official","path":"config.py","file_url":"https://github.com/cornell-zhang/heurigym/blob/HEAD/config.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6fcd80e8413e9b07"}},{"code_sha256_prefix":"0a0b8626cedf1ab8","entry":"evaluate","repo":"cornell-zhang/heurigym","repo_kind":"official","path":"global_routing/baseline/run_baseline.py","file_url":"https://github.com/cornell-zhang/heurigym/blob/HEAD/global_routing/baseline/run_baseline.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0a0b8626cedf1ab8"}},{"code_sha256_prefix":"45282c499a87b003","entry":"generate_summary_table","repo":"cornell-zhang/heurigym","repo_kind":"official","path":"llm_solver_agent.py","file_url":"https://github.com/cornell-zhang/heurigym/blob/HEAD/llm_solver_agent.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"45282c499a87b003"}},{"code_sha256_prefix":"4ab4ed9eea06dabc","entry":"get_model_pricing","repo":"cornell-zhang/heurigym","repo_kind":"official","path":"config.py","file_url":"https://github.com/cornell-zhang/heurigym/blob/HEAD/config.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4ab4ed9eea06dabc"}},{"code_sha256_prefix":"5bd4fd26f4dc8ebb","entry":"greedy","repo":"cornell-zhang/heurigym","repo_kind":"official","path":"egraph_extraction/baseline/greedy_baseline.py","file_url":"https://github.com/cornell-zhang/heurigym/blob/HEAD/egraph_extraction/baseline/greedy_baseline.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5bd4fd26f4dc8ebb"}},{"code_sha256_prefix":"ba48fb31f2879a84","entry":"load_graph","repo":"cornell-zhang/heurigym","repo_kind":"official","path":"egraph_extraction/program/verifier.py","file_url":"https://github.com/cornell-zhang/heurigym/blob/HEAD/egraph_extraction/program/verifier.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ba48fb31f2879a84"}},{"code_sha256_prefix":"f6a39d9a77e36206","entry":"load_solution","repo":"cornell-zhang/heurigym","repo_kind":"official","path":"egraph_extraction/program/verifier.py","file_url":"https://github.com/cornell-zhang/heurigym/blob/HEAD/egraph_extraction/program/verifier.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f6a39d9a77e36206"}},{"code_sha256_prefix":"0d1082ba17362e02","entry":"normalize_score","repo":"cornell-zhang/heurigym","repo_kind":"official","path":"egraph_extraction/program/metric.py","file_url":"https://github.com/cornell-zhang/heurigym/blob/HEAD/egraph_extraction/program/metric.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0d1082ba17362e02"}},{"code_sha256_prefix":"9f4010a5778070cb","entry":"parse_best_results","repo":"cornell-zhang/heurigym","repo_kind":"official","path":"llm_solver_agent.py","file_url":"https://github.com/cornell-zhang/heurigym/blob/HEAD/llm_solver_agent.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9f4010a5778070cb"}},{"code_sha256_prefix":"77d809abe68c246b","entry":"read_instance","repo":"cornell-zhang/heurigym","repo_kind":"official","path":"crew_pairing/program/utils.py","file_url":"https://github.com/cornell-zhang/heurigym/blob/HEAD/crew_pairing/program/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"77d809abe68c246b"}},{"code_sha256_prefix":"94646147e106b4ed","entry":"run_instantgr","repo":"cornell-zhang/heurigym","repo_kind":"official","path":"global_routing/baseline/run_baseline.py","file_url":"https://github.com/cornell-zhang/heurigym/blob/HEAD/global_routing/baseline/run_baseline.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"94646147e106b4ed"}},{"code_sha256_prefix":"4fa724a25bba66d7","entry":"verify","repo":"cornell-zhang/heurigym","repo_kind":"official","path":"egraph_extraction/program/verifier.py","file_url":"https://github.com/cornell-zhang/heurigym/blob/HEAD/egraph_extraction/program/verifier.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4fa724a25bba66d7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}