{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/autoplanbench-automatically-generating","title":"Automating the Generation of Prompts for LLM-based Action Choice in PDDL Planning","arxiv_id":"2311.09830","date":"2023-11-16","proceeding":null,"authors":["Katharina Stein","Daniel Fišer","Jörg Hoffmann","Alexander Koller"],"abstract":"Large language models (LLMs) have revolutionized a large variety of NLP tasks. An active debate is to what extent they can do reasoning and planning. Prior work has assessed the latter in the specific context of PDDL planning, based on manually converting three PDDL domains into natural language (NL) prompts. Here we automate this conversion step, showing how to leverage an LLM to automatically generate NL prompts from PDDL input. Our automatically generated NL prompts result in similar LLM-planning performance as the previous manually generated ones. Beyond this, the automation enables us to run much larger experiments, providing for the first time a broad evaluation of LLM planning performance in PDDL. Our NL prompts yield better performance than PDDL prompts and simple template-based NL prompts. Compared to symbolic planners, LLM planning lags far behind; but in some domains, our best LLM configuration scales up further than A$^\\star$ using LM-cut.","url_abs":"https://arxiv.org/abs/2311.09830v4","url_pdf":"https://arxiv.org/pdf/2311.09830v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"autoplanbench-automatically-generating","repo_url":"https://github.com/minecraft-saar/autoplanbench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.09830","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.09830"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/minecraft-saar/autoplanbench","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8181146756c066b7","entry":"generate_domain_setup_steps","repo":"minecraft-saar/autoplanbench","repo_kind":"official","path":"generate_steps.py","file_url":"https://github.com/minecraft-saar/autoplanbench/blob/HEAD/generate_steps.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8181146756c066b7"}},{"code_sha256_prefix":"738fc3018864823b","entry":"generate_domain_setup_steps_pddl","repo":"minecraft-saar/autoplanbench","repo_kind":"official","path":"generate_steps.py","file_url":"https://github.com/minecraft-saar/autoplanbench/blob/HEAD/generate_steps.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"738fc3018864823b"}},{"code_sha256_prefix":"fa4da7d0fef47041","entry":"generate_domain_setup_steps_templates","repo":"minecraft-saar/autoplanbench","repo_kind":"official","path":"generate_steps.py","file_url":"https://github.com/minecraft-saar/autoplanbench/blob/HEAD/generate_steps.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fa4da7d0fef47041"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}