{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/can-multimodal-large-language-models","title":"Can Multimodal Large Language Models Understand Spatial Relations?","arxiv_id":"2505.19015","date":"2025-05-25","proceeding":null,"authors":["Jingping Liu","Ziyan Liu","Zhedong Cen","Yan Zhou","Yinan Zou","Weiyan Zhang","Haiyun Jiang","Tong Ruan"],"abstract":"Spatial relation reasoning is a crucial task for multimodal large language models (MLLMs) to understand the objective world. However, current benchmarks have issues like relying on bounding boxes, ignoring perspective substitutions, or allowing questions to be answered using only the model's prior knowledge without image understanding. To address these issues, we introduce SpatialMQA, a human-annotated spatial relation reasoning benchmark based on COCO2017, which enables MLLMs to focus more on understanding images in the objective world. To ensure data quality, we design a well-tailored annotation procedure, resulting in SpatialMQA consisting of 5,392 samples. Based on this benchmark, a series of closed- and open-source MLLMs are implemented and the results indicate that the current state-of-the-art MLLM achieves only 48.14% accuracy, far below the human-level accuracy of 98.40%. Extensive experimental analyses are also conducted, suggesting the future research directions. The benchmark and codes are available at https://github.com/ziyan-xiaoyu/SpatialMQA.git.","url_abs":"https://arxiv.org/abs/2505.19015v1","url_pdf":"https://arxiv.org/pdf/2505.19015v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"can-multimodal-large-language-models","repo_url":"https://github.com/ziyan-xiaoyu/spatialmqa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":null,"task_name":"Relation"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2505.19015","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.19015"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ziyan-xiaoyu/spatialmqa","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6f5887ee4a95cd41","entry":"call_gpt4","repo":"ziyan-xiaoyu/spatialmqa","repo_kind":"official","path":"Code/close_models/gpt4_1_shot.py","file_url":"https://github.com/ziyan-xiaoyu/spatialmqa/blob/HEAD/Code/close_models/gpt4_1_shot.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6f5887ee4a95cd41"}},{"code_sha256_prefix":"b60704e5254f6958","entry":"encode_image","repo":"ziyan-xiaoyu/spatialmqa","repo_kind":"official","path":"Code/close_models/gpt4_1_shot.py","file_url":"https://github.com/ziyan-xiaoyu/spatialmqa/blob/HEAD/Code/close_models/gpt4_1_shot.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b60704e5254f6958"}},{"code_sha256_prefix":"adea9261c6c4fc88","entry":"fetch_image_content","repo":"ziyan-xiaoyu/spatialmqa","repo_kind":"official","path":"Code/close_models/gemini_0_shot.py","file_url":"https://github.com/ziyan-xiaoyu/spatialmqa/blob/HEAD/Code/close_models/gemini_0_shot.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"adea9261c6c4fc88"}},{"code_sha256_prefix":"3ff3d5cbf9ba7fd0","entry":"fetch_image_content","repo":"ziyan-xiaoyu/spatialmqa","repo_kind":"official","path":"Code/close_models/gpt4_1_shot.py","file_url":"https://github.com/ziyan-xiaoyu/spatialmqa/blob/HEAD/Code/close_models/gpt4_1_shot.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3ff3d5cbf9ba7fd0"}},{"code_sha256_prefix":"b53f578739839a48","entry":"pop_exist","repo":"ziyan-xiaoyu/spatialmqa","repo_kind":"official","path":"Code/close_models/gemini_2_shot_random.py","file_url":"https://github.com/ziyan-xiaoyu/spatialmqa/blob/HEAD/Code/close_models/gemini_2_shot_random.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b53f578739839a48"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}