{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/clip-under-the-microscope-a-fine-grained","title":"CLIP Under the Microscope: A Fine-Grained Analysis of Multi-Object Representation","arxiv_id":"2502.19842","date":"2025-02-27","proceeding":"CVPR 2025 1","authors":["Reza Abbasi","Ali Nazari","Aminreza Sefid","Mohammadali Banayeeanzade","Mohammad Hossein Rohban","Mahdieh Soleymani Baghshah"],"abstract":"Contrastive Language-Image Pre-training (CLIP) models excel in zero-shot classification, yet face challenges in complex multi-object scenarios. This study offers a comprehensive analysis of CLIP's limitations in these contexts using a specialized dataset, ComCO, designed to evaluate CLIP's encoders in diverse multi-object scenarios. Our findings reveal significant biases: the text encoder prioritizes first-mentioned objects, and the image encoder favors larger objects. Through retrieval and classification tasks, we quantify these biases across multiple CLIP variants and trace their origins to CLIP's training process, supported by analyses of the LAION dataset and training progression. Our image-text matching experiments show substantial performance drops when object size or token order changes, underscoring CLIP's instability with rephrased but semantically similar captions. Extending this to longer captions and text-to-image models like Stable Diffusion, we demonstrate how prompt order influences object prominence in generated images. For more details and access to our dataset and analysis code, visit our project repository: https://clip-analysis.github.io.","url_abs":"https://arxiv.org/abs/2502.19842v1","url_pdf":"https://arxiv.org/pdf/2502.19842v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"clip-under-the-microscope-a-fine-grained","repo_url":"https://github.com/clip-oscope/clip-oscope","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-text-matching","task_name":"Image-text matching"},{"task_slug":"object","task_name":"Object"},{"task_slug":"text-matching","task_name":"Text Matching"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"},{"task_slug":null,"task_name":"zero-shot-classification"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[{"slug":"simco-comco","name":"simco-comco","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2502.19842","atlas_url":"https://app.syntology.ai/?focus=2502.19842","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.19842"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/clip-oscope/clip-oscope","reach":null}],"summary":{"ran_violates":2,"ran_honours":2,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"2fe7dbe4e29f9fce","entry":"acc","repo":"clip-oscope/clip-oscope","repo_kind":"official","path":"Image_Encoder_Experiments/IOC.py","file_url":"https://github.com/clip-oscope/clip-oscope/blob/HEAD/Image_Encoder_Experiments/IOC.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2fe7dbe4e29f9fce"}},{"code_sha256_prefix":"e0a06ded5d4f6c3c","entry":"box_cxcywh_to_xyxy","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"e0a06ded5d4f6c3c"}},{"code_sha256_prefix":"1fccf780546b4856","entry":"check_end_of_sampling","repo":"clip-oscope/clip-oscope","repo_kind":"official","path":"Text_Encoder_Experiments/TOC_ComCO_CLIP.py","file_url":"https://github.com/clip-oscope/clip-oscope/blob/HEAD/Text_Encoder_Experiments/TOC_ComCO_CLIP.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1fccf780546b4856"}},{"code_sha256_prefix":"2db86df34e872f85","entry":"extract_image_features","repo":"clip-oscope/clip-oscope","repo_kind":"official","path":"Image_Encoder_Experiments/IOC.py","file_url":"https://github.com/clip-oscope/clip-oscope/blob/HEAD/Image_Encoder_Experiments/IOC.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2db86df34e872f85"}},{"code_sha256_prefix":"6b7d0e028122c634","entry":"get_phrases_from_posmap","repo":"clip-oscope/clip-oscope","repo_kind":"official","path":"Preparations/Lang-Sam/Groundino-laion.py","file_url":"https://github.com/clip-oscope/clip-oscope/blob/HEAD/Preparations/Lang-Sam/Groundino-laion.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6b7d0e028122c634"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}