{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/urbench-a-comprehensive-benchmark-for","title":"UrBench: A Comprehensive Benchmark for Evaluating Large Multimodal Models in Multi-View Urban Scenarios","arxiv_id":"2408.17267","date":"2024-08-30","proceeding":null,"authors":["Baichuan Zhou","Haote Yang","Dairong Chen","Junyan Ye","Tianyi Bai","Jinhua Yu","Songyang Zhang","Dahua Lin","Conghui He","Weijia Li"],"abstract":"Recent evaluations of Large Multimodal Models (LMMs) have explored their capabilities in various domains, with only few benchmarks specifically focusing on urban environments. Moreover, existing urban benchmarks have been limited to evaluating LMMs with basic region-level urban tasks under singular views, leading to incomplete evaluations of LMMs' abilities in urban environments. To address these issues, we present UrBench, a comprehensive benchmark designed for evaluating LMMs in complex multi-view urban scenarios. UrBench contains 11.6K meticulously curated questions at both region-level and role-level that cover 4 task dimensions: Geo-Localization, Scene Reasoning, Scene Understanding, and Object Understanding, totaling 14 task types. In constructing UrBench, we utilize data from existing datasets and additionally collect data from 11 cities, creating new annotations using a cross-view detection-matching method. With these images and annotations, we then integrate LMM-based, rule-based, and human-based methods to construct large-scale high-quality questions. Our evaluations on 21 LMMs show that current LMMs struggle in the urban environments in several aspects. Even the best performing GPT-4o lags behind humans in most tasks, ranging from simple tasks such as counting to complex tasks such as orientation, localization and object attribute recognition, with an average performance gap of 17.4%. Our benchmark also reveals that LMMs exhibit inconsistent behaviors with different urban views, especially with respect to understanding cross-view relations. UrBench datasets and benchmark results will be publicly available at https://opendatalab.github.io/UrBench/.","url_abs":"https://arxiv.org/abs/2408.17267v2","url_pdf":"https://arxiv.org/pdf/2408.17267v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"urbench-a-comprehensive-benchmark-for","repo_url":"https://github.com/opendatalab/urbench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"scene-understanding","task_name":"Scene Understanding"},{"task_slug":"geo-localization","task_name":"geo-localization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2408.17267","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.17267"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/opendatalab/urbench","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":6,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"38465d12e62ee88d","entry":"concatenate_images_horizontal","repo":"opendatalab/urbench","repo_kind":"official","path":"lmms_eval/tasks/urbench/utils_group_img.py","file_url":"https://github.com/opendatalab/urbench/blob/HEAD/lmms_eval/tasks/urbench/utils_group_img.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"38465d12e62ee88d"}},{"code_sha256_prefix":"5096e10842a99b97","entry":"construct_prompt","repo":"opendatalab/urbench","repo_kind":"official","path":"lmms_eval/tasks/urbench/utils.py","file_url":"https://github.com/opendatalab/urbench/blob/HEAD/lmms_eval/tasks/urbench/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5096e10842a99b97"}},{"code_sha256_prefix":"ea06eaae4fc1eaf0","entry":"hash_args","repo":"opendatalab/urbench","repo_kind":"official","path":"lmms_eval/api/model.py","file_url":"https://github.com/opendatalab/urbench/blob/HEAD/lmms_eval/api/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ea06eaae4fc1eaf0"}},{"code_sha256_prefix":"9fee17e752dc0a05","entry":"parse_options","repo":"opendatalab/urbench","repo_kind":"official","path":"lmms_eval/tasks/urbench/utils.py","file_url":"https://github.com/opendatalab/urbench/blob/HEAD/lmms_eval/tasks/urbench/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9fee17e752dc0a05"}},{"code_sha256_prefix":"74386ef186ff5c64","entry":"remove_none_pattern","repo":"opendatalab/urbench","repo_kind":"official","path":"lmms_eval/logging_utils.py","file_url":"https://github.com/opendatalab/urbench/blob/HEAD/lmms_eval/logging_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"74386ef186ff5c64"}},{"code_sha256_prefix":"8089a4a42eaa2452","entry":"replace_images_tokens","repo":"opendatalab/urbench","repo_kind":"official","path":"lmms_eval/tasks/urbench/utils.py","file_url":"https://github.com/opendatalab/urbench/blob/HEAD/lmms_eval/tasks/urbench/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8089a4a42eaa2452"}},{"code_sha256_prefix":"557122e7dd47b0b6","entry":"resize_image_height","repo":"opendatalab/urbench","repo_kind":"official","path":"lmms_eval/tasks/urbench/utils_group_img.py","file_url":"https://github.com/opendatalab/urbench/blob/HEAD/lmms_eval/tasks/urbench/utils_group_img.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"557122e7dd47b0b6"}},{"code_sha256_prefix":"395620f44f4978bb","entry":"add_order_label","repo":"opendatalab/urbench","repo_kind":"official","path":"lmms_eval/tasks/urbench/utils_group_img.py","file_url":"https://github.com/opendatalab/urbench/blob/HEAD/lmms_eval/tasks/urbench/utils_group_img.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"395620f44f4978bb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}