{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/planning-in-strawberry-fields-evaluating-and","title":"Planning in Strawberry Fields: Evaluating and Improving the Planning and Scheduling Capabilities of LRM o1","arxiv_id":"2410.02162","date":"2024-10-03","proceeding":null,"authors":["Karthik Valmeekam","Kaya Stechly","Atharva Gundawar","Subbarao Kambhampati"],"abstract":"The ability to plan a course of action that achieves a desired state of affairs has long been considered a core competence of intelligent agents and has been an integral part of AI research since its inception. With the advent of large language models (LLMs), there has been considerable interest in the question of whether or not they possess such planning abilities, but -- despite the slew of new private and open source LLMs since GPT3 -- progress has remained slow. OpenAI claims that their recent o1 (Strawberry) model has been specifically constructed and trained to escape the normal limitations of autoregressive LLMs -- making it a new kind of model: a Large Reasoning Model (LRM). In this paper, we evaluate the planning capabilities of two LRMs (o1-preview and o1-mini) on both planning and scheduling benchmarks. We see that while o1 does seem to offer significant improvements over autoregressive LLMs, this comes at a steep inference cost, while still failing to provide any guarantees over what it generates. We also show that combining o1 models with external verifiers -- in a so-called LRM-Modulo system -- guarantees the correctness of the combined system's output while further improving performance.","url_abs":"https://arxiv.org/abs/2410.02162v1","url_pdf":"https://arxiv.org/pdf/2410.02162v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"planning-in-strawberry-fields-evaluating-and","repo_url":"https://github.com/karthikv792/gpt-plan-benchmark","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"planning-in-strawberry-fields-evaluating-and","repo_url":"https://github.com/karthikv792/llms-planning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"scheduling","task_name":"Scheduling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2410.02162","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.02162"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/karthikv792/llms-planning","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/karthikv792/gpt-plan-benchmark","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":8,"unverified":1},"by_repo_kind":{"listed":{"samples":9,"ran":8,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8221b37cc068b36a","entry":"fill_template","repo":"karthikv792/gpt-plan-benchmark","repo_kind":"listed","path":"plan-bench/utils/pddl_to_text.py","file_url":"https://github.com/karthikv792/gpt-plan-benchmark/blob/HEAD/plan-bench/utils/pddl_to_text.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8221b37cc068b36a"}},{"code_sha256_prefix":"52b78255f2b9a159","entry":"generate_from_bloom","repo":"karthikv792/gpt-plan-benchmark","repo_kind":"listed","path":"plan-bench/utils/llm_utils.py","file_url":"https://github.com/karthikv792/gpt-plan-benchmark/blob/HEAD/plan-bench/utils/llm_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"52b78255f2b9a159"}},{"code_sha256_prefix":"5a1377db1f3d97a5","entry":"get_data","repo":"karthikv792/llms-planning","repo_kind":"listed","path":"llm_planning_analysis/stats_generation.py","file_url":"https://github.com/karthikv792/llms-planning/blob/HEAD/llm_planning_analysis/stats_generation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a1377db1f3d97a5"}},{"code_sha256_prefix":"b088e7075b491add","entry":"get_sorted","repo":"karthikv792/gpt-plan-benchmark","repo_kind":"listed","path":"plan-bench/utils/pddl_to_text.py","file_url":"https://github.com/karthikv792/gpt-plan-benchmark/blob/HEAD/plan-bench/utils/pddl_to_text.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b088e7075b491add"}},{"code_sha256_prefix":"799968ced0b94957","entry":"parse_problem","repo":"karthikv792/gpt-plan-benchmark","repo_kind":"listed","path":"plan-bench/utils/pddl_to_text.py","file_url":"https://github.com/karthikv792/gpt-plan-benchmark/blob/HEAD/plan-bench/utils/pddl_to_text.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"799968ced0b94957"}},{"code_sha256_prefix":"232066a31b3b39f1","entry":"random_mapping","repo":"karthikv792/gpt-plan-benchmark","repo_kind":"listed","path":"plan-bench/obfuscator.py","file_url":"https://github.com/karthikv792/gpt-plan-benchmark/blob/HEAD/plan-bench/obfuscator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"232066a31b3b39f1"}},{"code_sha256_prefix":"e56a33cbda477581","entry":"send_query","repo":"karthikv792/gpt-plan-benchmark","repo_kind":"listed","path":"plan-bench/utils/llm_utils.py","file_url":"https://github.com/karthikv792/gpt-plan-benchmark/blob/HEAD/plan-bench/utils/llm_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e56a33cbda477581"}},{"code_sha256_prefix":"57a2407f329edb96","entry":"store_predicates","repo":"karthikv792/gpt-plan-benchmark","repo_kind":"listed","path":"plan-bench/model_parser/parser_new.py","file_url":"https://github.com/karthikv792/gpt-plan-benchmark/blob/HEAD/plan-bench/model_parser/parser_new.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"57a2407f329edb96"}},{"code_sha256_prefix":"51fd8b7713cc3071","entry":"random_mapping","repo":"karthikv792/llms-planning","repo_kind":"listed","path":"llm_planning_analysis/obfuscator.py","file_url":"https://github.com/karthikv792/llms-planning/blob/HEAD/llm_planning_analysis/obfuscator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"51fd8b7713cc3071"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}