{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/planetarium-a-rigorous-benchmark-for","title":"Planetarium: A Rigorous Benchmark for Translating Text to Structured Planning Languages","arxiv_id":"2407.03321","date":"2024-07-03","proceeding":null,"authors":["Max Zuo","Francisco Piedrahita Velez","Xiaochen Li","Michael L. Littman","Stephen H. Bach"],"abstract":"Many recent works have explored using language models for planning problems. One line of research focuses on translating natural language descriptions of planning tasks into structured planning languages, such as the planning domain definition language (PDDL). While this approach is promising, accurately measuring the quality of generated PDDL code continues to pose significant challenges. First, generated PDDL code is typically evaluated using planning validators that check whether the problem can be solved with a planner. This method is insufficient because a language model might generate valid PDDL code that does not align with the natural language description of the task. Second, existing evaluation sets often have natural language descriptions of the planning task that closely resemble the ground truth PDDL, reducing the challenge of the task. To bridge this gap, we introduce \\benchmarkName, a benchmark designed to evaluate language models' ability to generate PDDL code from natural language descriptions of planning tasks. We begin by creating a PDDL equivalence algorithm that rigorously evaluates the correctness of PDDL code generated by language models by flexibly comparing it against a ground truth PDDL. Then, we present a dataset of $132,037$ text-to-PDDL pairs across 13 different tasks, with varying levels of difficulty. Finally, we evaluate several API-access and open-weight language models that reveal this task's complexity. For example, $87.6\\%$ of the PDDL problem descriptions generated by GPT-4o are syntactically parseable, $82.2\\%$ are valid, solve-able problems, but only $35.1\\%$ are semantically correct, highlighting the need for a more rigorous benchmark for this problem.","url_abs":"https://arxiv.org/abs/2407.03321v1","url_pdf":"https://arxiv.org/pdf/2407.03321v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"planetarium-a-rigorous-benchmark-for","repo_url":"https://github.com/batsresearch/planetarium","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":null,"task_name":"valid"}],"methods":[],"datasets_introduced":[{"slug":"planetarium","name":"Planetarium","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2407.03321","atlas_url":"https://app.syntology.ai/?focus=2407.03321","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.03321"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/batsresearch/planetarium","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"unverified":2},"by_repo_kind":{"official":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f86334af13a45187","entry":"plan","repo":"batsresearch/planetarium","repo_kind":"official","path":"planetarium/downward.py","file_url":"https://github.com/batsresearch/planetarium/blob/HEAD/planetarium/downward.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f86334af13a45187"}},{"code_sha256_prefix":"02ae2bc034213037","entry":"validate","repo":"batsresearch/planetarium","repo_kind":"official","path":"planetarium/downward.py","file_url":"https://github.com/batsresearch/planetarium/blob/HEAD/planetarium/downward.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"02ae2bc034213037"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}