{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/colorswap-a-color-and-word-order-dataset-for","title":"ColorSwap: A Color and Word Order Dataset for Multimodal Evaluation","arxiv_id":"2402.04492","date":"2024-02-07","proceeding":null,"authors":["Jirayu Burapacheep","Ishan Gaur","Agam Bhatia","Tristan Thrush"],"abstract":"This paper introduces the ColorSwap dataset, designed to assess and improve the proficiency of multimodal models in matching objects with their colors. The dataset is comprised of 2,000 unique image-caption pairs, grouped into 1,000 examples. Each example includes a caption-image pair, along with a ``color-swapped'' pair. We follow the Winoground schema: the two captions in an example have the same words, but the color words have been rearranged to modify different objects. The dataset was created through a novel blend of automated caption and image generation with humans in the loop. We evaluate image-text matching (ITM) and visual language models (VLMs) and find that even the latest ones are still not robust at this task. GPT-4V and LLaVA score 72% and 42% on our main VLM metric, although they may improve with more advanced prompting techniques. On the main ITM metric, contrastive models such as CLIP and SigLIP perform close to chance (at 12% and 30%, respectively), although the non-contrastive BLIP ITM model is stronger (87%). We also find that finetuning on fewer than 2,000 examples yields significant performance gains on this out-of-distribution word-order understanding task. The dataset is here: https://github.com/Top34051/colorswap and here: https://huggingface.co/datasets/stanfordnlp/colorswap.","url_abs":"https://arxiv.org/abs/2402.04492v2","url_pdf":"https://arxiv.org/pdf/2402.04492v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"colorswap-a-color-and-word-order-dataset-for","repo_url":"https://github.com/top34051/colorswap","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"image-text-matching","task_name":"Image-text matching"},{"task_slug":"text-matching","task_name":"Text Matching"}],"methods":[{"method_slug":"blip","method_name":"BLIP"},{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2402.04492","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.04492"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/top34051/colorswap","reach":null}],"summary":{"ran_draft_wrong":1,"ran_violates":2},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c6fa6303a18a9165","entry":"extract_options_text","repo":"top34051/colorswap","repo_kind":"official","path":"vlm_eval.py","file_url":"https://github.com/top34051/colorswap/blob/HEAD/vlm_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c6fa6303a18a9165"}},{"code_sha256_prefix":"ad4894a8640cf39f","entry":"is_option_image","repo":"top34051/colorswap","repo_kind":"official","path":"vlm_eval.py","file_url":"https://github.com/top34051/colorswap/blob/HEAD/vlm_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad4894a8640cf39f"}},{"code_sha256_prefix":"4e15a7bb18d68024","entry":"is_option_text","repo":"top34051/colorswap","repo_kind":"official","path":"vlm_eval.py","file_url":"https://github.com/top34051/colorswap/blob/HEAD/vlm_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4e15a7bb18d68024"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}