{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/what-s-up-with-vision-language-models","title":"What's \"up\" with vision-language models? Investigating their struggle with spatial reasoning","arxiv_id":"2310.19785","date":"2023-10-30","proceeding":null,"authors":["Amita Kamath","Jack Hessel","Kai-Wei Chang"],"abstract":"Recent vision-language (VL) models are powerful, but can they reliably distinguish \"right\" from \"left\"? We curate three new corpora to quantify model comprehension of such basic spatial relations. These tests isolate spatial reasoning more precisely than existing datasets like VQAv2, e.g., our What'sUp benchmark contains sets of photographs varying only the spatial relations of objects, keeping their identity fixed (see Figure 1: models must comprehend not only the usual case of a dog under a table, but also, the same dog on top of the same table). We evaluate 18 VL models, finding that all perform poorly, e.g., BLIP finetuned on VQAv2, which nears human parity on VQAv2, achieves 56% accuracy on our benchmarks vs. humans at 99%. We conclude by studying causes of this surprising behavior, finding: 1) that popular vision-language pretraining corpora like LAION-2B contain little reliable data for learning spatial relationships; and 2) that basic modeling interventions like up-weighting preposition-containing instances or fine-tuning on our corpora are not sufficient to address the challenges our benchmarks pose. We are hopeful that these corpora will facilitate further research, and we release our data and code at https://github.com/amitakamath/whatsup_vlms.","url_abs":"https://arxiv.org/abs/2310.19785v1","url_pdf":"https://arxiv.org/pdf/2310.19785v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"what-s-up-with-vision-language-models","repo_url":"https://github.com/amitakamath/whatsup_vlms","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"what-s-up-with-vision-language-models","repo_url":"https://github.com/shiqichen17/adaptvis","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"spatial-reasoning","task_name":"Spatial Reasoning"}],"methods":[{"method_slug":"blip","method_name":"BLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.19785","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.19785"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amitakamath/whatsup_vlms","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shiqichen17/adaptvis","reach":null}],"summary":{"ran_draft_wrong":2,"ran":3,"unverified":4},"by_repo_kind":{"official":{"samples":7,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"373a7df152e5a1a5","entry":"apply_rotary_pos_emb","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"373a7df152e5a1a5"}},{"code_sha256_prefix":"d52f072444282bb2","entry":"get_image_perturb_fn","repo":"amitakamath/whatsup_vlms","repo_kind":"official","path":"dataset_zoo/perturbations.py","file_url":"https://github.com/amitakamath/whatsup_vlms/blob/HEAD/dataset_zoo/perturbations.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d52f072444282bb2"}},{"code_sha256_prefix":"0d760ca65749dc14","entry":"pre_caption","repo":"amitakamath/whatsup_vlms","repo_kind":"official","path":"dataset_zoo/retrieval.py","file_url":"https://github.com/amitakamath/whatsup_vlms/blob/HEAD/dataset_zoo/retrieval.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d760ca65749dc14"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"29d83052b338a1f7","entry":"shuffle_rows","repo":"amitakamath/whatsup_vlms","repo_kind":"official","path":"dataset_zoo/perturbations.py","file_url":"https://github.com/amitakamath/whatsup_vlms/blob/HEAD/dataset_zoo/perturbations.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"29d83052b338a1f7"}},{"code_sha256_prefix":"1bd954c1ba107cb9","entry":"blip_decoder","repo":"amitakamath/whatsup_vlms","repo_kind":"official","path":"model_zoo/blip_utils/blip.py","file_url":"https://github.com/amitakamath/whatsup_vlms/blob/HEAD/model_zoo/blip_utils/blip.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1bd954c1ba107cb9"}},{"code_sha256_prefix":"16c013eb492b2d8c","entry":"blip_feature_extractor","repo":"amitakamath/whatsup_vlms","repo_kind":"official","path":"model_zoo/blip_utils/blip.py","file_url":"https://github.com/amitakamath/whatsup_vlms/blob/HEAD/model_zoo/blip_utils/blip.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"16c013eb492b2d8c"}},{"code_sha256_prefix":"f837c334ddfa8af4","entry":"create_vit","repo":"amitakamath/whatsup_vlms","repo_kind":"official","path":"model_zoo/blip_utils/blip.py","file_url":"https://github.com/amitakamath/whatsup_vlms/blob/HEAD/model_zoo/blip_utils/blip.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f837c334ddfa8af4"}},{"code_sha256_prefix":"5c8ae0e86a1836f6","entry":"get_text_perturb_fn","repo":"amitakamath/whatsup_vlms","repo_kind":"official","path":"dataset_zoo/perturbations.py","file_url":"https://github.com/amitakamath/whatsup_vlms/blob/HEAD/dataset_zoo/perturbations.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5c8ae0e86a1836f6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}