{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/clip-behaves-like-a-bag-of-words-model-cross","title":"CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modally","arxiv_id":"2502.03566","date":"2025-02-05","proceeding":null,"authors":["Darina Koishigarina","Arnas Uselis","Seong Joon Oh"],"abstract":"CLIP (Contrastive Language-Image Pretraining) has become a popular choice for various downstream tasks. However, recent studies have questioned its ability to represent compositional concepts effectively. These works suggest that CLIP often acts like a bag-of-words (BoW) model, interpreting images and text as sets of individual concepts without grasping the structural relationships. In particular, CLIP struggles to correctly bind attributes to their corresponding objects when multiple objects are present in an image or text. In this work, we investigate why CLIP exhibits this BoW-like behavior. We find that the correct attribute-object binding information is already present in individual text and image modalities. Instead, the issue lies in the cross-modal alignment, which relies on cosine similarity. To address this, we propose Linear Attribute Binding CLIP or LABCLIP. It applies a linear transformation to text embeddings before computing cosine similarity. This approach significantly improves CLIP's ability to bind attributes to correct objects, thereby enhancing its compositional understanding. The code is available at https://github.com/kdariina/CLIP-not-BoW-unimodally.","url_abs":"https://arxiv.org/abs/2502.03566v2","url_pdf":"https://arxiv.org/pdf/2502.03566v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"clip-behaves-like-a-bag-of-words-model-cross","repo_url":"https://github.com/kdariina/clip-not-bow-unimodally","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"cross-modal-alignment","task_name":"cross-modal alignment"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2502.03566","atlas_url":"https://app.syntology.ai/?focus=2502.03566","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.03566"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kdariina/clip-not-bow-unimodally","reach":null}],"summary":{"ran_honours":1,"ran_draft_wrong":2},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"df665c35e2490fbf","entry":"calculate_accuracy","repo":"kdariina/clip-not-bow-unimodally","repo_kind":"official","path":"probing/probing_models.py","file_url":"https://github.com/kdariina/clip-not-bow-unimodally/blob/HEAD/probing/probing_models.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"df665c35e2490fbf"}},{"code_sha256_prefix":"3eb3380f163cc578","entry":"constrastive_loss","repo":"kdariina/clip-not-bow-unimodally","repo_kind":"official","path":"alignment/learning_alignment.py","file_url":"https://github.com/kdariina/clip-not-bow-unimodally/blob/HEAD/alignment/learning_alignment.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3eb3380f163cc578"}},{"code_sha256_prefix":"756c1cea908d428b","entry":"constrastive_loss_with_negatives","repo":"kdariina/clip-not-bow-unimodally","repo_kind":"official","path":"alignment/learning_alignment.py","file_url":"https://github.com/kdariina/clip-not-bow-unimodally/blob/HEAD/alignment/learning_alignment.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"756c1cea908d428b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}