{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2608-03882","title":"MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning","arxiv_id":"2608.03882","date":"2026-08-04","proceeding":null,"authors":["Martin Böckling","Elizaveta Nosova","Heiko Paulheim","Andreea Iana"],"abstract":"Geospatial reasoning, i.e., computing distances, containment, and other spatial relations over real-world entities, is central to navigation and logistics, yet large language models (LLMs) struggle with the required geometric and topological computation despite storing considerable geographic knowledge. Existing benchmarks localize these failures only partially: they are synthetic or smallscale, largely monolingual, and offer limited control over geographic coverage. We introduce MultiGlobeQA, a multilingual benchmark of 46,060 question-answer pairs spanning 14 spatial-function families and 15 answer formats, with execution-based ground truth over three knowledge graphs. It covers 201 countries and territories via income- and density-stratified sampling, with parallel questions in English and 16 additional high- and low-resource languages. Across parametric, reasoning, and agentic settings, LLMs collapse on tasks requiring grid indexing and shape computation, while topological relations and directions fare best. Retrieval and tool use yield considerable gains, yet performance plateaus below two thirds even when gold facts are supplied, indicating that computation, not access to knowledge, is the bottleneck. Models also underperform on low-income regions, a gap that gold facts widen rather than close.","url_abs":"https://arxiv.org/abs/2608.03882","url_pdf":"https://arxiv.org/pdf/2608.03882","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2608.03882","atlas_url":null,"mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2608.03882"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/andreeaiana/MultiGlobeQA","reach":null}],"summary":{"ran":5,"unverified":4},"by_repo_kind":{"found_in_text":{"samples":9,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2aeb019cb8885b78","entry":"discover_benchmark_shards","repo":"andreeaiana/MultiGlobeQA","repo_kind":"found_in_text","path":"benchmark_models/io.py","file_url":"https://github.com/andreeaiana/MultiGlobeQA/blob/HEAD/benchmark_models/io.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2aeb019cb8885b78"}},{"code_sha256_prefix":"fbde67cdfd8cc9bb","entry":"format_estimate_table","repo":"andreeaiana/MultiGlobeQA","repo_kind":"found_in_text","path":"benchmark_models/cost_estimator.py","file_url":"https://github.com/andreeaiana/MultiGlobeQA/blob/HEAD/benchmark_models/cost_estimator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fbde67cdfd8cc9bb"}},{"code_sha256_prefix":"2b5a48968230a28d","entry":"load_experiment_config","repo":"andreeaiana/MultiGlobeQA","repo_kind":"found_in_text","path":"benchmark_models/config.py","file_url":"https://github.com/andreeaiana/MultiGlobeQA/blob/HEAD/benchmark_models/config.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2b5a48968230a28d"}},{"code_sha256_prefix":"e1baa7e8dbb5a725","entry":"parse_bool_or_none","repo":"andreeaiana/MultiGlobeQA","repo_kind":"found_in_text","path":"benchmark_models/base.py","file_url":"https://github.com/andreeaiana/MultiGlobeQA/blob/HEAD/benchmark_models/base.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e1baa7e8dbb5a725"}},{"code_sha256_prefix":"1478051fde069a2e","entry":"resolve_image_path","repo":"andreeaiana/MultiGlobeQA","repo_kind":"found_in_text","path":"benchmark_models/image_paths.py","file_url":"https://github.com/andreeaiana/MultiGlobeQA/blob/HEAD/benchmark_models/image_paths.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1478051fde069a2e"}},{"code_sha256_prefix":"dbc95bb775b8f7fb","entry":"discover_gold_shards","repo":"andreeaiana/MultiGlobeQA","repo_kind":"found_in_text","path":"benchmark_models/io.py","file_url":"https://github.com/andreeaiana/MultiGlobeQA/blob/HEAD/benchmark_models/io.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dbc95bb775b8f7fb"}},{"code_sha256_prefix":"06ee56ae37aa8831","entry":"format_raw","repo":"andreeaiana/MultiGlobeQA","repo_kind":"found_in_text","path":"benchmark_models/oracle.py","file_url":"https://github.com/andreeaiana/MultiGlobeQA/blob/HEAD/benchmark_models/oracle.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"06ee56ae37aa8831"}},{"code_sha256_prefix":"bbc9a4b776a843be","entry":"format_structured","repo":"andreeaiana/MultiGlobeQA","repo_kind":"found_in_text","path":"benchmark_models/oracle.py","file_url":"https://github.com/andreeaiana/MultiGlobeQA/blob/HEAD/benchmark_models/oracle.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bbc9a4b776a843be"}},{"code_sha256_prefix":"278418097249bddc","entry":"simplify_wkt","repo":"andreeaiana/MultiGlobeQA","repo_kind":"found_in_text","path":"benchmark_models/oracle_simplify.py","file_url":"https://github.com/andreeaiana/MultiGlobeQA/blob/HEAD/benchmark_models/oracle_simplify.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"278418097249bddc"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,885 of the 9,623 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9623,"papers_checked":6885},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}