{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/citybench-evaluating-the-capabilities-of","title":"CityBench: Evaluating the Capabilities of Large Language Models for Urban Tasks","arxiv_id":"2406.13945","date":"2024-06-20","proceeding":null,"authors":["Jie Feng","Jun Zhang","Tianhui Liu","Xin Zhang","Tianjian Ouyang","Junbo Yan","Yuwei Du","Siqi Guo","Yong Li"],"abstract":"Recently, large language models (LLMs) with extensive general knowledge and powerful reasoning abilities have seen rapid development and widespread application. A systematic and reliable evaluation of LLMs or vision-language model (VLMs) is a crucial step in applying and developing them for various fields. There have been some early explorations about the usability of LLMs for limited urban tasks, but a systematic and scalable evaluation benchmark is still lacking. The challenge in constructing a systematic evaluation benchmark for urban research lies in the diversity of urban data, the complexity of application scenarios and the highly dynamic nature of the urban environment. In this paper, we design CityBench, an interactive simulator based evaluation platform, as the first systematic benchmark for evaluating the capabilities of LLMs for diverse tasks in urban research. First, we build CityData to integrate the diverse urban data and CitySimu to simulate fine-grained urban dynamics. Based on CityData and CitySimu, we design 8 representative urban tasks in 2 categories of perception-understanding and decision-making as the CityBench. With extensive results from 30 well-known LLMs and VLMs in 13 cities around the world, we find that advanced LLMs and VLMs can achieve competitive performance in diverse urban tasks requiring commonsense and semantic understanding abilities, e.g., understanding the human dynamics and semantic inference of urban images. Meanwhile, they fail to solve the challenging urban tasks requiring professional knowledge and high-level reasoning abilities, e.g., geospatial prediction and traffic control task. These observations provide valuable perspectives for utilizing and developing LLMs in the future. Codes are openly accessible via https://github.com/tsinghua-fib-lab/CityBench.","url_abs":"https://arxiv.org/abs/2406.13945v2","url_pdf":"https://arxiv.org/pdf/2406.13945v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"citybench-evaluating-the-capabilities-of","repo_url":"https://github.com/tsinghua-fib-lab/citybench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"general-knowledge","task_name":"General Knowledge"},{"task_slug":"human-dynamics","task_name":"Human Dynamics"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.13945","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.13945"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tsinghua-fib-lab/citybench","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":16,"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"official":{"samples":23,"ran":17,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ed0baa8be5a2be8e","entry":"bd09_to_gcj02","repo":"tsinghua-fib-lab/citybench","repo_kind":"official","path":"citybench/outdoor_navigation/coordTransform_utils.py","file_url":"https://github.com/tsinghua-fib-lab/citybench/blob/HEAD/citybench/outdoor_navigation/coordTransform_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed0baa8be5a2be8e"}},{"code_sha256_prefix":"3dd52b32d8265d75","entry":"convert_to_12_hour_clock","repo":"tsinghua-fib-lab/citybench","repo_kind":"official","path":"citybench/mobility_prediction/run_eval.py","file_url":"https://github.com/tsinghua-fib-lab/citybench/blob/HEAD/citybench/mobility_prediction/run_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3dd52b32d8265d75"}},{"code_sha256_prefix":"afe3f93eb4daddb2","entry":"dir_all_dis","repo":"tsinghua-fib-lab/citybench","repo_kind":"official","path":"citybench/geoqa/utils.py","file_url":"https://github.com/tsinghua-fib-lab/citybench/blob/HEAD/citybench/geoqa/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"afe3f93eb4daddb2"}},{"code_sha256_prefix":"fbde55d407dfbc1f","entry":"format_example","repo":"tsinghua-fib-lab/citybench","repo_kind":"official","path":"citybench/geoqa/run_eval.py","file_url":"https://github.com/tsinghua-fib-lab/citybench/blob/HEAD/citybench/geoqa/run_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fbde55d407dfbc1f"}},{"code_sha256_prefix":"6011c01be1bc239c","entry":"gcj02_to_bd09","repo":"tsinghua-fib-lab/citybench","repo_kind":"official","path":"citybench/outdoor_navigation/coordTransform_utils.py","file_url":"https://github.com/tsinghua-fib-lab/citybench/blob/HEAD/citybench/outdoor_navigation/coordTransform_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6011c01be1bc239c"}},{"code_sha256_prefix":"9745e19e1e4d5474","entry":"gen_options","repo":"tsinghua-fib-lab/citybench","repo_kind":"official","path":"citybench/geoqa/utils.py","file_url":"https://github.com/tsinghua-fib-lab/citybench/blob/HEAD/citybench/geoqa/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9745e19e1e4d5474"}},{"code_sha256_prefix":"901473d434f03b50","entry":"get_acc1_f1","repo":"tsinghua-fib-lab/citybench","repo_kind":"official","path":"citybench/mobility_prediction/metrics.py","file_url":"https://github.com/tsinghua-fib-lab/citybench/blob/HEAD/citybench/mobility_prediction/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"901473d434f03b50"}},{"code_sha256_prefix":"00ff8bb07bb6500f","entry":"get_day","repo":"tsinghua-fib-lab/citybench","repo_kind":"official","path":"citybench/mobility_prediction/data_gen.py","file_url":"https://github.com/tsinghua-fib-lab/citybench/blob/HEAD/citybench/mobility_prediction/data_gen.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"00ff8bb07bb6500f"}},{"code_sha256_prefix":"cd1e4cb179726aa5","entry":"get_is_correct","repo":"tsinghua-fib-lab/citybench","repo_kind":"official","path":"citybench/mobility_prediction/metrics.py","file_url":"https://github.com/tsinghua-fib-lab/citybench/blob/HEAD/citybench/mobility_prediction/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cd1e4cb179726aa5"}},{"code_sha256_prefix":"bdfec11b2815eede","entry":"get_is_correct10","repo":"tsinghua-fib-lab/citybench","repo_kind":"official","path":"citybench/mobility_prediction/metrics.py","file_url":"https://github.com/tsinghua-fib-lab/citybench/blob/HEAD/citybench/mobility_prediction/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bdfec11b2815eede"}},{"code_sha256_prefix":"aafdf0d32c0ac451","entry":"get_nearby_pois","repo":"tsinghua-fib-lab/citybench","repo_kind":"official","path":"citybench/geoqa/data_gen.py","file_url":"https://github.com/tsinghua-fib-lab/citybench/blob/HEAD/citybench/geoqa/data_gen.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aafdf0d32c0ac451"}},{"code_sha256_prefix":"7a47bc523075e6d4","entry":"get_result","repo":"tsinghua-fib-lab/citybench","repo_kind":"official","path":"citybench/geoqa/metrics.py","file_url":"https://github.com/tsinghua-fib-lab/citybench/blob/HEAD/citybench/geoqa/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7a47bc523075e6d4"}},{"code_sha256_prefix":"5642b97cf4bef5f0","entry":"get_time","repo":"tsinghua-fib-lab/citybench","repo_kind":"official","path":"citybench/mobility_prediction/data_gen.py","file_url":"https://github.com/tsinghua-fib-lab/citybench/blob/HEAD/citybench/mobility_prediction/data_gen.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5642b97cf4bef5f0"}},{"code_sha256_prefix":"d45560a97cfd6154","entry":"get_user_data","repo":"tsinghua-fib-lab/citybench","repo_kind":"official","path":"citybench/mobility_prediction/run_eval.py","file_url":"https://github.com/tsinghua-fib-lab/citybench/blob/HEAD/citybench/mobility_prediction/run_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d45560a97cfd6154"}},{"code_sha256_prefix":"4189bccf9c32f695","entry":"task_files_adaption","repo":"tsinghua-fib-lab/citybench","repo_kind":"official","path":"citybench/geoqa/run_eval.py","file_url":"https://github.com/tsinghua-fib-lab/citybench/blob/HEAD/citybench/geoqa/run_eval.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4189bccf9c32f695"}},{"code_sha256_prefix":"9b36c7f6d6007d25","entry":"task_files_adaption","repo":"tsinghua-fib-lab/citybench","repo_kind":"official","path":"citybench/geoqa/utils.py","file_url":"https://github.com/tsinghua-fib-lab/citybench/blob/HEAD/citybench/geoqa/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9b36c7f6d6007d25"}},{"code_sha256_prefix":"e67cb36ee1894010","entry":"wgs84_to_gcj02","repo":"tsinghua-fib-lab/citybench","repo_kind":"official","path":"citybench/outdoor_navigation/coordTransform_utils.py","file_url":"https://github.com/tsinghua-fib-lab/citybench/blob/HEAD/citybench/outdoor_navigation/coordTransform_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e67cb36ee1894010"}},{"code_sha256_prefix":"673d2b1bf01f10aa","entry":"coord_conv","repo":"tsinghua-fib-lab/citybench","repo_kind":"official","path":"citybench/outdoor_navigation/crawler_baidu.py","file_url":"https://github.com/tsinghua-fib-lab/citybench/blob/HEAD/citybench/outdoor_navigation/crawler_baidu.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"673d2b1bf01f10aa"}},{"code_sha256_prefix":"17a3a8f86db336ac","entry":"coord_conv_back","repo":"tsinghua-fib-lab/citybench","repo_kind":"official","path":"citybench/outdoor_navigation/crawler_baidu.py","file_url":"https://github.com/tsinghua-fib-lab/citybench/blob/HEAD/citybench/outdoor_navigation/crawler_baidu.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"17a3a8f86db336ac"}},{"code_sha256_prefix":"3205b9f39f5f2e0c","entry":"gps_gen","repo":"tsinghua-fib-lab/citybench","repo_kind":"official","path":"citybench/geoqa/data_gen.py","file_url":"https://github.com/tsinghua-fib-lab/citybench/blob/HEAD/citybench/geoqa/data_gen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3205b9f39f5f2e0c"}},{"code_sha256_prefix":"74042dda0a06339b","entry":"landmark_gen","repo":"tsinghua-fib-lab/citybench","repo_kind":"official","path":"citybench/geoqa/data_gen.py","file_url":"https://github.com/tsinghua-fib-lab/citybench/blob/HEAD/citybench/geoqa/data_gen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"74042dda0a06339b"}},{"code_sha256_prefix":"a9cc1853676c97f4","entry":"loc_to_sid","repo":"tsinghua-fib-lab/citybench","repo_kind":"official","path":"citybench/outdoor_navigation/crawler_baidu.py","file_url":"https://github.com/tsinghua-fib-lab/citybench/blob/HEAD/citybench/outdoor_navigation/crawler_baidu.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a9cc1853676c97f4"}},{"code_sha256_prefix":"8b39129551bb06a0","entry":"match_pos","repo":"tsinghua-fib-lab/citybench","repo_kind":"official","path":"citybench/mobility_prediction/data_gen.py","file_url":"https://github.com/tsinghua-fib-lab/citybench/blob/HEAD/citybench/mobility_prediction/data_gen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8b39129551bb06a0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}