{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/flex-travelplanner-a-benchmark-for-flexible","title":"Flex-TravelPlanner: A Benchmark for Flexible Planning with Language Agents","arxiv_id":"2506.04649","date":"2025-06-05","proceeding":null,"authors":["Juhyun Oh","Eunsu Kim","Alice Oh"],"abstract":"Real-world planning problems require constant adaptation to changing requirements and balancing of competing constraints. However, current benchmarks for evaluating LLMs' planning capabilities primarily focus on static, single-turn scenarios. We introduce Flex-TravelPlanner, a benchmark that evaluates language models' ability to reason flexibly in dynamic planning scenarios. Building on the TravelPlanner dataset~\\citep{xie2024travelplanner}, we introduce two novel evaluation settings: (1) sequential constraint introduction across multiple turns, and (2) scenarios with explicitly prioritized competing constraints. Our analysis of GPT-4o and Llama 3.1 70B reveals several key findings: models' performance on single-turn tasks poorly predicts their ability to adapt plans across multiple turns; constraint introduction order significantly affects performance; and models struggle with constraint prioritization, often incorrectly favoring newly introduced lower priority preferences over existing higher-priority constraints. These findings highlight the importance of evaluating LLMs in more realistic, dynamic planning scenarios and suggest specific directions for improving model performance on complex planning tasks. The code and dataset for our framework are publicly available at https://github.com/juhyunohh/FlexTravelBench.","url_abs":"https://arxiv.org/abs/2506.04649v1","url_pdf":"https://arxiv.org/pdf/2506.04649v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"flex-travelplanner-a-benchmark-for-flexible","repo_url":"https://github.com/juhyunohh/flextravelbench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"llama","method_name":"LLaMA"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2506.04649","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.04649"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/juhyunohh/flextravelbench","reach":null}],"summary":{"ran_draft_wrong":3,"ran_violates":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"d1c8e2418591e73c","entry":"extract_constraints","repo":"juhyunohh/flextravelbench","repo_kind":"official","path":"implement/agents/dataset_generate.py","file_url":"https://github.com/juhyunohh/flextravelbench/blob/HEAD/implement/agents/dataset_generate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d1c8e2418591e73c"}},{"code_sha256_prefix":"0022f4a8b6ab88e1","entry":"filter_dataset_by_condition","repo":"juhyunohh/flextravelbench","repo_kind":"official","path":"implement/evaluate.py","file_url":"https://github.com/juhyunohh/flextravelbench/blob/HEAD/implement/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0022f4a8b6ab88e1"}},{"code_sha256_prefix":"5ea19cd29432fd85","entry":"should_process_example","repo":"juhyunohh/flextravelbench","repo_kind":"official","path":"implement/evaluate.py","file_url":"https://github.com/juhyunohh/flextravelbench/blob/HEAD/implement/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5ea19cd29432fd85"}},{"code_sha256_prefix":"a6af92ee3e1e0287","entry":"update_local_constraints","repo":"juhyunohh/flextravelbench","repo_kind":"official","path":"implement/agents/dataset_generate.py","file_url":"https://github.com/juhyunohh/flextravelbench/blob/HEAD/implement/agents/dataset_generate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a6af92ee3e1e0287"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}