{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/probing-conceptual-understanding-of-large","title":"Probing Conceptual Understanding of Large Visual-Language Models","arxiv_id":"2304.03659","date":"2023-04-07","proceeding":null,"authors":["Madeline Schiappa","Raiyaan Abdullah","Shehreen Azad","Jared Claypoole","Michael Cogswell","Ajay Divakaran","Yogesh Rawat"],"abstract":"In recent years large visual-language (V+L) models have achieved great success in various downstream tasks. However, it is not well studied whether these models have a conceptual grasp of the visual content. In this work we focus on conceptual understanding of these large V+L models. To facilitate this study, we propose novel benchmarking datasets for probing three different aspects of content understanding, 1) \\textit{relations}, 2) \\textit{composition}, and 3) \\textit{context}. Our probes are grounded in cognitive science and help determine if a V+L model can, for example, determine if snow garnished with a man is implausible, or if it can identify beach furniture by knowing it is located on a beach. We experimented with many recent state-of-the-art V+L models and observe that these models mostly \\textit{fail to demonstrate} a conceptual understanding. This study reveals several interesting insights such as that \\textit{cross-attention} helps learning conceptual understanding, and that CNNs are better with \\textit{texture and patterns}, while Transformers are better at \\textit{color and shape}. We further utilize some of these insights and investigate a \\textit{simple finetuning technique} that rewards the three conceptual understanding measures with promising initial results. The proposed benchmarks will drive the community to delve deeper into conceptual understanding and foster advancements in the capabilities of large V+L models. The code and dataset is available at: \\url{https://tinyurl.com/vlm-robustness}","url_abs":"https://arxiv.org/abs/2304.03659v3","url_pdf":"https://arxiv.org/pdf/2304.03659v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"probing-conceptual-understanding-of-large","repo_url":"https://github.com/Maddy12/UnderstandingVisualTextModels","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"fail","method_name":"fail"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2304.03659","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2304.03659"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Maddy12/UnderstandingVisualTextModels","reach":null}],"summary":{"ran_fixture":1,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"cca051584677b059","entry":"average_precision","repo":"Maddy12/UnderstandingVisualTextModels","repo_kind":"official","path":"core/eval_context_understanding.py","file_url":"https://github.com/Maddy12/UnderstandingVisualTextModels/blob/HEAD/core/eval_context_understanding.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cca051584677b059"}},{"code_sha256_prefix":"da1cfae6a5deaa69","entry":"make_comparisons","repo":"Maddy12/UnderstandingVisualTextModels","repo_kind":"official","path":"core/eval_context_understanding.py","file_url":"https://github.com/Maddy12/UnderstandingVisualTextModels/blob/HEAD/core/eval_context_understanding.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"da1cfae6a5deaa69"}},{"code_sha256_prefix":"4088ef65bb0d79f1","entry":"eval_flava","repo":"Maddy12/UnderstandingVisualTextModels","repo_kind":"official","path":"core/eval_relation_understanding.py","file_url":"https://github.com/Maddy12/UnderstandingVisualTextModels/blob/HEAD/core/eval_relation_understanding.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4088ef65bb0d79f1"}},{"code_sha256_prefix":"6cd4b4c43a746200","entry":"eval_flava","repo":"Maddy12/UnderstandingVisualTextModels","repo_kind":"official","path":"core/eval_compositional_understanding.py","file_url":"https://github.com/Maddy12/UnderstandingVisualTextModels/blob/HEAD/core/eval_compositional_understanding.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6cd4b4c43a746200"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}