{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vision-language-models-are-blind","title":"Vision language models are blind: Failing to translate detailed visual features into words","arxiv_id":"2407.06581","date":"2024-07-09","proceeding":null,"authors":["Pooyan Rahmanzadehgervi","Logan Bolton","Mohammad Reza Taesiri","Anh Totti Nguyen"],"abstract":"While large language models with vision capabilities (VLMs), e.g., GPT-4o and Gemini 1.5 Pro, score high on many vision-understanding benchmarks, they are still struggling with low-level vision tasks that are easy to humans. Specifically, on BlindTest, our suite of 7 very simple tasks, including identifying (a) whether two circles overlap; (b) how many times two lines intersect; (c) which letter is being circled in a word; and (d) the number of circles in an Olympic-like logo, four state-of-the-art VLMs are only 58.07% accurate on average. Claude 3.5 Sonnet performs the best at 77.84% accuracy, far from the human expected accuracy of 100%. Across different image resolutions and line widths, VLMs including slow-thinking models consistently struggle with those tasks that require precise spatial information when geometric primitives overlap or are close. Yet, VLMs perform at near-100% accuracy when much more space is added to separate shapes and letters. Linear probing experiments show that vision encoders contain sufficient visual information to solve BlindTest and that language models fail to decode this information into correct answers. Code and data are at: https://vlmsareblind.github.io","url_abs":"https://arxiv.org/abs/2407.06581v6","url_pdf":"https://arxiv.org/pdf/2407.06581v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vision-language-models-are-blind","repo_url":"https://github.com/anguyen8/vision-llms-are-blind","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.06581","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.06581"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/anguyen8/vision-llms-are-blind","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"ran_draft_wrong":2,"ran_honours":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3c76e52815c5401d","entry":"repeat_kv","repo":"anguyen8/vision-llms-are-blind","repo_kind":"official","path":"src/LinearProbe/Phi3.5/modeling_phi3_v.py","file_url":"https://github.com/anguyen8/vision-llms-are-blind/blob/HEAD/src/LinearProbe/Phi3.5/modeling_phi3_v.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c76e52815c5401d"}},{"code_sha256_prefix":"583539efd6fd01fb","entry":"apply_rotary_pos_emb","repo":"anguyen8/vision-llms-are-blind","repo_kind":"official","path":"src/LinearProbe/Phi3.5/modeling_phi3_v.py","file_url":"https://github.com/anguyen8/vision-llms-are-blind/blob/HEAD/src/LinearProbe/Phi3.5/modeling_phi3_v.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"583539efd6fd01fb"}},{"code_sha256_prefix":"9b3c1cb391672ccb","entry":"load_image","repo":"anguyen8/vision-llms-are-blind","repo_kind":"official","path":"src/LinearProbe/LLaVA-NeXT/predict.py","file_url":"https://github.com/anguyen8/vision-llms-are-blind/blob/HEAD/src/LinearProbe/LLaVA-NeXT/predict.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9b3c1cb391672ccb"}},{"code_sha256_prefix":"e03d53ba9d4f9ae5","entry":"rotate_half","repo":"anguyen8/vision-llms-are-blind","repo_kind":"official","path":"src/LinearProbe/Phi3.5/modeling_phi3_v.py","file_url":"https://github.com/anguyen8/vision-llms-are-blind/blob/HEAD/src/LinearProbe/Phi3.5/modeling_phi3_v.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e03d53ba9d4f9ae5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}