{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/locationreasoner-evaluating-llms-on-real","title":"LocationReasoner: Evaluating LLMs on Real-World Site Selection Reasoning","arxiv_id":"2506.13841","date":"2025-06-16","proceeding":null,"authors":["Miho Koda","Yu Zheng","Ruixian Ma","Mingyang Sun","Devesh Pansare","Fabio Duarte","Paolo Santi"],"abstract":"Recent advances in large language models (LLMs), particularly those enhanced through reinforced post-training, have demonstrated impressive reasoning capabilities, as exemplified by models such as OpenAI o1 and DeepSeek-R1. However, these capabilities are predominantly benchmarked on domains like mathematical problem solving and code generation -- leaving open the question of whether such reasoning skills generalize to complex, real-world scenarios. In this paper, we introduce LocationReasoner, a benchmark designed to evaluate LLMs' reasoning abilities in the context of real-world site selection, where models must identify feasible locations by reasoning over diverse and complicated spatial, environmental, and logistical constraints. The benchmark comprises over 300 carefully crafted queries of varying difficulty levels, supported by a sandbox environment with in-house tools for constraint-based location search. Extensive evaluations reveal that state-of-the-art reasoning models offer limited improvement over their non-reasoning predecessors in real-world contexts, with even the latest OpenAI o4 model failing on 30% of site selection tasks. Moreover, agentic strategies such as ReAct and Reflexion often suffer from over-reasoning, leading to worse outcomes than direct code-generation prompting. With key limitations of LLMs in holistic and non-linear reasoning highlighted, we release LocationReasoner to foster the development of LLMs and agents capable of robust, grounded reasoning in real-world decision-making tasks. Codes and data for our benchmark are available at https://github.com/miho-koda/LocationReasoner.","url_abs":"https://arxiv.org/abs/2506.13841v1","url_pdf":"https://arxiv.org/pdf/2506.13841v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"locationreasoner-evaluating-llms-on-real","repo_url":"https://github.com/miho-koda/locationreasoner","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"mathematical-problem-solving","task_name":"Mathematical Problem-Solving"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2506.13841","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.13841"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/miho-koda/LocationReasoner","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/miho-koda/locationreasoner","reach":{"status":"ok"}}],"summary":{"unverified":11},"by_repo_kind":{"official":{"samples":11,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"00fda2704b022424","entry":"clean_code_string","repo":"miho-koda/LocationReasoner","repo_kind":"official","path":"code/executors/code_task_executor.py","file_url":"https://github.com/miho-koda/LocationReasoner/blob/HEAD/code/executors/code_task_executor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"00fda2704b022424"}},{"code_sha256_prefix":"69bd58f5c3d58a5d","entry":"clean_code_string","repo":"miho-koda/LocationReasoner","repo_kind":"official","path":"code/executors/complex_task_executor.py","file_url":"https://github.com/miho-koda/LocationReasoner/blob/HEAD/code/executors/complex_task_executor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"69bd58f5c3d58a5d"}},{"code_sha256_prefix":"a8bf6e1766e37ae5","entry":"clean_code_string","repo":"miho-koda/locationreasoner","repo_kind":"official","path":"code/executors/run_harder.py","file_url":"https://github.com/miho-koda/locationreasoner/blob/HEAD/code/executors/run_harder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a8bf6e1766e37ae5"}},{"code_sha256_prefix":"1c85fe06d9836515","entry":"combine_zone_sets","repo":"miho-koda/LocationReasoner","repo_kind":"official","path":"code/executors/complex_task_executor.py","file_url":"https://github.com/miho-koda/LocationReasoner/blob/HEAD/code/executors/complex_task_executor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1c85fe06d9836515"}},{"code_sha256_prefix":"ab16d163be8d585c","entry":"compare_zones","repo":"miho-koda/LocationReasoner","repo_kind":"official","path":"code/executors/claude_code_executor.py","file_url":"https://github.com/miho-koda/LocationReasoner/blob/HEAD/code/executors/claude_code_executor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ab16d163be8d585c"}},{"code_sha256_prefix":"474e1d327bf447c9","entry":"extract_python_code","repo":"miho-koda/LocationReasoner","repo_kind":"official","path":"code/core/router.py","file_url":"https://github.com/miho-koda/LocationReasoner/blob/HEAD/code/core/router.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"474e1d327bf447c9"}},{"code_sha256_prefix":"71f61c43e0e66b0c","entry":"get_logistics_path","repo":"miho-koda/LocationReasoner","repo_kind":"official","path":"code/executors/city_batch_executor.py","file_url":"https://github.com/miho-koda/LocationReasoner/blob/HEAD/code/executors/city_batch_executor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"71f61c43e0e66b0c"}},{"code_sha256_prefix":"5b813380b449d9ec","entry":"load_config","repo":"miho-koda/LocationReasoner","repo_kind":"official","path":"code/config_utils.py","file_url":"https://github.com/miho-koda/LocationReasoner/blob/HEAD/code/config_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5b813380b449d9ec"}},{"code_sha256_prefix":"288e94e2e03d35d5","entry":"make_runner_script","repo":"miho-koda/LocationReasoner","repo_kind":"official","path":"code/executors/run_harder.py","file_url":"https://github.com/miho-koda/LocationReasoner/blob/HEAD/code/executors/run_harder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"288e94e2e03d35d5"}},{"code_sha256_prefix":"eadcd2834f643642","entry":"resolve_paths","repo":"miho-koda/LocationReasoner","repo_kind":"official","path":"code/config_utils.py","file_url":"https://github.com/miho-koda/LocationReasoner/blob/HEAD/code/config_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eadcd2834f643642"}},{"code_sha256_prefix":"afdd65ea18e2ba86","entry":"run_claude_code","repo":"miho-koda/LocationReasoner","repo_kind":"official","path":"code/executors/claude_code_executor.py","file_url":"https://github.com/miho-koda/LocationReasoner/blob/HEAD/code/executors/claude_code_executor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"afdd65ea18e2ba86"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}