{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/coreval-a-comprehensive-and-objective","title":"CHOICE: Benchmarking the Remote Sensing Capabilities of Large Vision-Language Models","arxiv_id":"2411.18145","date":"2024-11-27","proceeding":null,"authors":["Xiao An","Jiaxing Sun","Zihan Gui","wei he"],"abstract":"The rapid advancement of Large Vision-Language Models (VLMs), both general-domain models and those specifically tailored for remote sensing, has demonstrated exceptional perception and reasoning capabilities in Earth observation tasks. However, a benchmark for systematically evaluating their capabilities in this domain is still lacking. To bridge this gap, we propose CHOICE, an extensive benchmark designed to objectively evaluate the hierarchical remote sensing capabilities of VLMs. Focusing on 2 primary capability dimensions essential to remote sensing: perception and reasoning, we further categorize 6 secondary dimensions and 23 leaf tasks to ensure a well-rounded assessment coverage. CHOICE guarantees the quality of all 10,507 problems through a rigorous process of data collection from 50 globally distributed cities, question construction and quality control. The newly curated data and the format of multiple-choice questions with definitive answers allow for an objective and straightforward performance assessment. Our evaluation of 3 proprietary and 21 open-source VLMs highlights their critical limitations within this specialized context. We hope that CHOICE will serve as a valuable resource and offer deeper insights into the challenges and potential of VLMs in the field of remote sensing. We will release CHOICE at https://github.com/ShawnAn-WHU/CHOICE.","url_abs":"https://arxiv.org/abs/2411.18145v3","url_pdf":"https://arxiv.org/pdf/2411.18145v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"coreval-a-comprehensive-and-objective","repo_url":"https://github.com/shawnan-whu/choice","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"earth-observation","task_name":"Earth Observation"},{"task_slug":"multiple-choice","task_name":"Multiple-choice"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2411.18145","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.18145"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shawnan-whu/choice","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"unverified":7},"by_repo_kind":{"official":{"samples":8,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bd5a05ce886da4ce","entry":"image_grid","repo":"shawnan-whu/choice","repo_kind":"official","path":"ms-swift/swift/aigc/diffusers/train_controlnet.py","file_url":"https://github.com/shawnan-whu/choice/blob/HEAD/ms-swift/swift/aigc/diffusers/train_controlnet.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd5a05ce886da4ce"}},{"code_sha256_prefix":"d647662c50b1c198","entry":"ce_loss_func","repo":"shawnan-whu/choice","repo_kind":"official","path":"ms-swift/swift/trainers/loss.py","file_url":"https://github.com/shawnan-whu/choice/blob/HEAD/ms-swift/swift/trainers/loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d647662c50b1c198"}},{"code_sha256_prefix":"2966975168bef617","entry":"check_model_is_id","repo":"shawnan-whu/choice","repo_kind":"official","path":"ms-swift/swift/hub/check_model.py","file_url":"https://github.com/shawnan-whu/choice/blob/HEAD/ms-swift/swift/hub/check_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2966975168bef617"}},{"code_sha256_prefix":"971d8096a21b52d4","entry":"get_regex_for_mm_default_lora","repo":"shawnan-whu/choice","repo_kind":"official","path":"ms-swift/swift/utils/module_mapping.py","file_url":"https://github.com/shawnan-whu/choice/blob/HEAD/ms-swift/swift/utils/module_mapping.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"971d8096a21b52d4"}},{"code_sha256_prefix":"1405282d2c0ce92b","entry":"import_model_class_from_model_name_or_path","repo":"shawnan-whu/choice","repo_kind":"official","path":"ms-swift/swift/aigc/diffusers/train_controlnet.py","file_url":"https://github.com/shawnan-whu/choice/blob/HEAD/ms-swift/swift/aigc/diffusers/train_controlnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1405282d2c0ce92b"}},{"code_sha256_prefix":"97f780e147e59a1a","entry":"lora_state_dict","repo":"shawnan-whu/choice","repo_kind":"official","path":"ms-swift/swift/tuners/lora_layers.py","file_url":"https://github.com/shawnan-whu/choice/blob/HEAD/ms-swift/swift/tuners/lora_layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"97f780e147e59a1a"}},{"code_sha256_prefix":"5cd7117272337c60","entry":"loss_scale_func","repo":"shawnan-whu/choice","repo_kind":"official","path":"ms-swift/swift/trainers/loss.py","file_url":"https://github.com/shawnan-whu/choice/blob/HEAD/ms-swift/swift/trainers/loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5cd7117272337c60"}},{"code_sha256_prefix":"800b1f7aa0ff47ba","entry":"register_loss_func","repo":"shawnan-whu/choice","repo_kind":"official","path":"ms-swift/swift/trainers/loss.py","file_url":"https://github.com/shawnan-whu/choice/blob/HEAD/ms-swift/swift/trainers/loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"800b1f7aa0ff47ba"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}