{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/do-vision-language-models-represent-space-and","title":"Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguities","arxiv_id":"2410.17385","date":"2024-10-22","proceeding":null,"authors":["Zheyuan Zhang","Fengyuan Hu","Jayjun Lee","Freda Shi","Parisa Kordjamshidi","Joyce Chai","Ziqiao Ma"],"abstract":"Spatial expressions in situated communication can be ambiguous, as their meanings vary depending on the frames of reference (FoR) adopted by speakers and listeners. While spatial language understanding and reasoning by vision-language models (VLMs) have gained increasing attention, potential ambiguities in these models are still under-explored. To address this issue, we present the COnsistent Multilingual Frame Of Reference Test (COMFORT), an evaluation protocol to systematically assess the spatial reasoning capabilities of VLMs. We evaluate nine state-of-the-art VLMs using COMFORT. Despite showing some alignment with English conventions in resolving ambiguities, our experiments reveal significant shortcomings of VLMs: notably, the models (1) exhibit poor robustness and consistency, (2) lack the flexibility to accommodate multiple FoRs, and (3) fail to adhere to language-specific or culture-specific conventions in cross-lingual tests, as English tends to dominate other languages. With a growing effort to align vision-language models with human cognitive intuitions, we call for more attention to the ambiguous nature and cross-cultural diversity of spatial reasoning.","url_abs":"https://arxiv.org/abs/2410.17385v2","url_pdf":"https://arxiv.org/pdf/2410.17385v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"do-vision-language-models-represent-space-and","repo_url":"https://github.com/sled-group/COMFORT","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"spatial-reasoning","task_name":"Spatial Reasoning"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[{"slug":"comfort","name":"COMFORT","full_name":"Consistent Multilingual Frame of Reference Test"}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.17385","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.17385"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sled-group/COMFORT","reach":{"status":"ok"}}],"summary":{"ran":11,"ran_draft_wrong":1,"ran_fixture":1,"unverified":4},"by_repo_kind":{"official":{"samples":17,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":17,"samples":[{"code_sha256_prefix":"119cf41aa9e4217f","entry":"accuracy_metric","repo":"sled-group/COMFORT","repo_kind":"official","path":"comfort_utils/accuracy_metric.py","file_url":"https://github.com/sled-group/COMFORT/blob/HEAD/comfort_utils/accuracy_metric.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"119cf41aa9e4217f"}},{"code_sha256_prefix":"8cf510a10c5f26c3","entry":"build_transform","repo":"sled-group/COMFORT","repo_kind":"official","path":"comfort_utils/model_utils/intern_vl.py","file_url":"https://github.com/sled-group/COMFORT/blob/HEAD/comfort_utils/model_utils/intern_vl.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8cf510a10c5f26c3"}},{"code_sha256_prefix":"e8579f20cd680a34","entry":"calculate_tokens_mean_attn","repo":"sled-group/COMFORT","repo_kind":"official","path":"comfort_utils/helper.py","file_url":"https://github.com/sled-group/COMFORT/blob/HEAD/comfort_utils/helper.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e8579f20cd680a34"}},{"code_sha256_prefix":"a721d26b902941ed","entry":"clean_latex_dataframe","repo":"sled-group/COMFORT","repo_kind":"official","path":"comfort_utils/convert_perspective_taking_results.py","file_url":"https://github.com/sled-group/COMFORT/blob/HEAD/comfort_utils/convert_perspective_taking_results.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a721d26b902941ed"}},{"code_sha256_prefix":"d0aec2c5a6c35427","entry":"convert_to_dict","repo":"sled-group/COMFORT","repo_kind":"official","path":"comfort_utils/convert_general_metrics_results.py","file_url":"https://github.com/sled-group/COMFORT/blob/HEAD/comfort_utils/convert_general_metrics_results.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d0aec2c5a6c35427"}},{"code_sha256_prefix":"d610b5eabe0c9db1","entry":"dynamic_preprocess","repo":"sled-group/COMFORT","repo_kind":"official","path":"comfort_utils/model_utils/intern_vl.py","file_url":"https://github.com/sled-group/COMFORT/blob/HEAD/comfort_utils/model_utils/intern_vl.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d610b5eabe0c9db1"}},{"code_sha256_prefix":"d3f911d03b9b9162","entry":"extract_table_data","repo":"sled-group/COMFORT","repo_kind":"official","path":"comfort_utils/convert_general_metrics_results.py","file_url":"https://github.com/sled-group/COMFORT/blob/HEAD/comfort_utils/convert_general_metrics_results.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d3f911d03b9b9162"}},{"code_sha256_prefix":"bd77f5f8067f18e9","entry":"find_closest_aspect_ratio","repo":"sled-group/COMFORT","repo_kind":"official","path":"comfort_utils/model_utils/intern_vl.py","file_url":"https://github.com/sled-group/COMFORT/blob/HEAD/comfort_utils/model_utils/intern_vl.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bd77f5f8067f18e9"}},{"code_sha256_prefix":"69b3b0492d912e11","entry":"find_index_of_image_tokens","repo":"sled-group/COMFORT","repo_kind":"official","path":"comfort_utils/helper.py","file_url":"https://github.com/sled-group/COMFORT/blob/HEAD/comfort_utils/helper.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"69b3b0492d912e11"}},{"code_sha256_prefix":"55c9cd0497a84208","entry":"grounding_enc_processor","repo":"sled-group/COMFORT","repo_kind":"official","path":"comfort_utils/model_utils/glamm_wrapper.py","file_url":"https://github.com/sled-group/COMFORT/blob/HEAD/comfort_utils/model_utils/glamm_wrapper.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"55c9cd0497a84208"}},{"code_sha256_prefix":"50dff60708886cd3","entry":"latex_to_dataframe","repo":"sled-group/COMFORT","repo_kind":"official","path":"comfort_utils/convert_perspective_taking_results.py","file_url":"https://github.com/sled-group/COMFORT/blob/HEAD/comfort_utils/convert_perspective_taking_results.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"50dff60708886cd3"}},{"code_sha256_prefix":"c8d3c86f905b4d9a","entry":"sort_dataframe_by_model","repo":"sled-group/COMFORT","repo_kind":"official","path":"comfort_utils/convert_perspective_taking_results.py","file_url":"https://github.com/sled-group/COMFORT/blob/HEAD/comfort_utils/convert_perspective_taking_results.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c8d3c86f905b4d9a"}},{"code_sha256_prefix":"99bc5e82093e6f29","entry":"sub_string_tokens_in_resp_token","repo":"sled-group/COMFORT","repo_kind":"official","path":"comfort_utils/model_utils/gpt_wrapper.py","file_url":"https://github.com/sled-group/COMFORT/blob/HEAD/comfort_utils/model_utils/gpt_wrapper.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"99bc5e82093e6f29"}},{"code_sha256_prefix":"2846873087c38293","entry":"accuracy_metric_object_frame","repo":"sled-group/COMFORT","repo_kind":"official","path":"comfort_utils/accuracy_metric.py","file_url":"https://github.com/sled-group/COMFORT/blob/HEAD/comfort_utils/accuracy_metric.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2846873087c38293"}},{"code_sha256_prefix":"0497e51919a7a2a6","entry":"accuracy_metric_old","repo":"sled-group/COMFORT","repo_kind":"official","path":"comfort_utils/accuracy_metric.py","file_url":"https://github.com/sled-group/COMFORT/blob/HEAD/comfort_utils/accuracy_metric.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0497e51919a7a2a6"}},{"code_sha256_prefix":"c68d30db2cd63b2a","entry":"convention_metric","repo":"sled-group/COMFORT","repo_kind":"official","path":"comfort_utils/convention_metric.py","file_url":"https://github.com/sled-group/COMFORT/blob/HEAD/comfort_utils/convention_metric.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c68d30db2cd63b2a"}},{"code_sha256_prefix":"c36a70c23b159d96","entry":"language_ambiguity_metric","repo":"sled-group/COMFORT","repo_kind":"official","path":"comfort_utils/language_ambiguity_metric.py","file_url":"https://github.com/sled-group/COMFORT/blob/HEAD/comfort_utils/language_ambiguity_metric.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c36a70c23b159d96"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}