{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rethinking-the-openness-of-clip","title":"Delving into the Openness of CLIP","arxiv_id":"2206.01986","date":"2022-06-04","proceeding":null,"authors":["Shuhuai Ren","Lei LI","Xuancheng Ren","Guangxiang Zhao","Xu sun"],"abstract":"Contrastive Language-Image Pre-training (CLIP) formulates image classification as an image-to-text matching task, i.e., matching images to the corresponding natural language descriptions instead of discrete category IDs. This allows for open-vocabulary visual recognition, where the model can recognize images from an open class set (also known as an open vocabulary) in a zero-shot manner. However, evaluating the openness of CLIP-like models is challenging, as the models are open to arbitrary vocabulary in theory, but their accuracy varies in practice. To address this, we resort to an incremental perspective to assess the openness through vocabulary expansions, and define extensibility to measure a model's ability to handle novel classes. Our evaluation shows that CLIP-like models are not truly open, and their performance deteriorates as the vocabulary expands. We further dissect the feature space of CLIP from the perspectives of representation alignment and uniformity. Our investigation reveals that the overestimation of openness is due to confusion among competing text features, rather than a failure to capture the similarity between image features and text features of novel classes. We hope that our investigation and analysis will facilitate future research on the CLIP openness issue.","url_abs":"https://arxiv.org/abs/2206.01986v3","url_pdf":"https://arxiv.org/pdf/2206.01986v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rethinking-the-openness-of-clip","repo_url":"https://github.com/lancopku/clip-openness","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-to-text","task_name":"Image to text"},{"task_slug":"text-matching","task_name":"Text Matching"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2206.01986","atlas_url":"https://app.syntology.ai/?focus=2206.01986","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.01986"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lancopku/clip-openness","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":3,"ran":2,"unverified":5},"by_repo_kind":{"official":{"samples":10,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"98f385d847636a3e","entry":"basic_clean","repo":"lancopku/clip-openness","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/lancopku/clip-openness/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"98f385d847636a3e"}},{"code_sha256_prefix":"d919ae32e5e4e616","entry":"get_pairs","repo":"lancopku/clip-openness","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/lancopku/clip-openness/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d919ae32e5e4e616"}},{"code_sha256_prefix":"42824ddb4bfbec48","entry":"metric_to_average","repo":"lancopku/clip-openness","repo_kind":"official","path":"clip-retrieval/clip_retrieval/clip_back.py","file_url":"https://github.com/lancopku/clip-openness/blob/HEAD/clip-retrieval/clip_retrieval/clip_back.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"42824ddb4bfbec48"}},{"code_sha256_prefix":"1472ce0e9f26f1c7","entry":"normalized","repo":"lancopku/clip-openness","repo_kind":"official","path":"clip-retrieval/clip_retrieval/clip_back.py","file_url":"https://github.com/lancopku/clip-openness/blob/HEAD/clip-retrieval/clip_retrieval/clip_back.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1472ce0e9f26f1c7"}},{"code_sha256_prefix":"9542161e9640b858","entry":"whitespace_clean","repo":"lancopku/clip-openness","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/lancopku/clip-openness/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9542161e9640b858"}},{"code_sha256_prefix":"c47aa9e5b049a11d","entry":"build_model","repo":"lancopku/clip-openness","repo_kind":"official","path":"clip/model.py","file_url":"https://github.com/lancopku/clip-openness/blob/HEAD/clip/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c47aa9e5b049a11d"}},{"code_sha256_prefix":"6055209be787c0d7","entry":"download_image","repo":"lancopku/clip-openness","repo_kind":"official","path":"clip-retrieval/clip_retrieval/clip_back.py","file_url":"https://github.com/lancopku/clip-openness/blob/HEAD/clip-retrieval/clip_retrieval/clip_back.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6055209be787c0d7"}},{"code_sha256_prefix":"5261e2f4314f6275","entry":"get_old_to_new_mapping","repo":"lancopku/clip-openness","repo_kind":"official","path":"clip-retrieval/clip_retrieval/ivf_metadata_ordering.py","file_url":"https://github.com/lancopku/clip-openness/blob/HEAD/clip-retrieval/clip_retrieval/ivf_metadata_ordering.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5261e2f4314f6275"}},{"code_sha256_prefix":"fbf8c0143d9c48e3","entry":"load","repo":"lancopku/clip-openness","repo_kind":"official","path":"clip/clip.py","file_url":"https://github.com/lancopku/clip-openness/blob/HEAD/clip/clip.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fbf8c0143d9c48e3"}},{"code_sha256_prefix":"c315c90997b8fc3c","entry":"search_to_new_ids","repo":"lancopku/clip-openness","repo_kind":"official","path":"clip-retrieval/clip_retrieval/ivf_metadata_ordering.py","file_url":"https://github.com/lancopku/clip-openness/blob/HEAD/clip-retrieval/clip_retrieval/ivf_metadata_ordering.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c315c90997b8fc3c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}