{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sugarcrepe-dataset-vision-language-model","title":"SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterations","arxiv_id":"2406.11171","date":"2024-06-17","proceeding":null,"authors":["Sri Harsha Dumpala","Aman Jaiswal","Chandramouli Sastry","Evangelos Milios","Sageev Oore","Hassan Sajjad"],"abstract":"Despite their remarkable successes, state-of-the-art large language models (LLMs), including vision-and-language models (VLMs) and unimodal language models (ULMs), fail to understand precise semantics. For example, semantically equivalent sentences expressed using different lexical compositions elicit diverging representations. The degree of this divergence and its impact on encoded semantics is not very well understood. In this paper, we introduce the SUGARCREPE++ dataset to analyze the sensitivity of VLMs and ULMs to lexical and semantic alterations. Each sample in SUGARCREPE++ dataset consists of an image and a corresponding triplet of captions: a pair of semantically equivalent but lexically different positive captions and one hard negative caption. This poses a 3-way semantic (in)equivalence problem to the language models. We comprehensively evaluate VLMs and ULMs that differ in architecture, pre-training objectives and datasets to benchmark the performance of SUGARCREPE++ dataset. Experimental results highlight the difficulties of VLMs in distinguishing between lexical and semantic variations, particularly in object attributes and spatial relations. Although VLMs with larger pre-training datasets, model sizes, and multiple pre-training objectives achieve better performance on SUGARCREPE++, there is a significant opportunity for improvement. We show that all the models which achieve better performance on compositionality datasets need not perform equally well on SUGARCREPE++, signifying that compositionality alone may not be sufficient for understanding semantic and lexical alterations. Given the importance of the property that the SUGARCREPE++ dataset targets, it serves as a new challenge to the vision-and-language community.","url_abs":"https://arxiv.org/abs/2406.11171v2","url_pdf":"https://arxiv.org/pdf/2406.11171v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sugarcrepe-dataset-vision-language-model","repo_url":"https://github.com/Sri-Harsha/scpp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"sensitivity","task_name":"Sensitivity"},{"task_slug":null,"task_name":"Triplet"}],"methods":[],"datasets_introduced":[{"slug":"sugarcrepe","name":"SugarCrepe++","full_name":"SugarCrepe++"}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.11171","atlas_url":"https://app.syntology.ai/?focus=2406.11171","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.11171"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Sri-Harsha/scpp","reach":null}],"summary":{"ran_draft_wrong":4,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7c6b861ca90aa566","entry":"clean_text","repo":"Sri-Harsha/scpp","repo_kind":"official","path":"generate_sugarcrepe_plus-mistral.py","file_url":"https://github.com/Sri-Harsha/scpp/blob/HEAD/generate_sugarcrepe_plus-mistral.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7c6b861ca90aa566"}},{"code_sha256_prefix":"cd7870b8098d6090","entry":"prepare_prompt","repo":"Sri-Harsha/scpp","repo_kind":"official","path":"generate_sugarcrepe_plus-mistral.py","file_url":"https://github.com/Sri-Harsha/scpp/blob/HEAD/generate_sugarcrepe_plus-mistral.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cd7870b8098d6090"}},{"code_sha256_prefix":"ce2c7f90934b82d4","entry":"prepare_prompt_to_generate_again","repo":"Sri-Harsha/scpp","repo_kind":"official","path":"generate_sugarcrepe_plus-mistral.py","file_url":"https://github.com/Sri-Harsha/scpp/blob/HEAD/generate_sugarcrepe_plus-mistral.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ce2c7f90934b82d4"}},{"code_sha256_prefix":"9bc8d3fbbbad0be2","entry":"setup_logger","repo":"Sri-Harsha/scpp","repo_kind":"official","path":"evaluate_ulms.py","file_url":"https://github.com/Sri-Harsha/scpp/blob/HEAD/evaluate_ulms.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9bc8d3fbbbad0be2"}},{"code_sha256_prefix":"181b4e43c1c329ce","entry":"summarize_results","repo":"Sri-Harsha/scpp","repo_kind":"official","path":"evaluate_ulms.py","file_url":"https://github.com/Sri-Harsha/scpp/blob/HEAD/evaluate_ulms.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"181b4e43c1c329ce"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}