{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/is-clip-the-main-roadblock-for-fine-grained","title":"Is CLIP the main roadblock for fine-grained open-world perception?","arxiv_id":"2404.03539","date":"2024-04-04","proceeding":null,"authors":["Lorenzo Bianchi","Fabio Carrara","Nicola Messina","Fabrizio Falchi"],"abstract":"Modern applications increasingly demand flexible computer vision models that adapt to novel concepts not encountered during training. This necessity is pivotal in emerging domains like extended reality, robotics, and autonomous driving, which require the ability to respond to open-world stimuli. A key ingredient is the ability to identify objects based on free-form textual queries defined at inference time - a task known as open-vocabulary object detection. Multimodal backbones like CLIP are the main enabling technology for current open-world perception solutions. Despite performing well on generic queries, recent studies highlighted limitations on the fine-grained recognition capabilities in open-vocabulary settings - i.e., for distinguishing subtle object features like color, shape, and material. In this paper, we perform a detailed examination of these open-vocabulary object recognition limitations to find the root cause. We evaluate the performance of CLIP, the most commonly used vision-language backbone, against a fine-grained object-matching benchmark, revealing interesting analogies between the limitations of open-vocabulary object detectors and their backbones. Experiments suggest that the lack of fine-grained understanding is caused by the poor separability of object characteristics in the CLIP latent space. Therefore, we try to understand whether fine-grained knowledge is present in CLIP embeddings but not exploited at inference time due, for example, to the unsuitability of the cosine similarity matching function, which may discard important object characteristics. Our preliminary experiments show that simple CLIP latent-space re-projections help separate fine-grained concepts, paving the way towards the development of backbones inherently able to process fine-grained details. The code for reproducing these experiments is available at https://github.com/lorebianchi98/FG-CLIP.","url_abs":"https://arxiv.org/abs/2404.03539v1","url_pdf":"https://arxiv.org/pdf/2404.03539v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"is-clip-the-main-roadblock-for-fine-grained","repo_url":"https://github.com/lorebianchi98/fg-clip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"is-clip-the-main-roadblock-for-fine-grained","repo_url":"https://github.com/lorebianchi98/fg-ovd","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"autonomous-driving","task_name":"Autonomous Driving"},{"task_slug":"novel-concepts","task_name":"Novel Concepts"},{"task_slug":"object","task_name":"Object"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"object-recognition","task_name":"Object Recognition"},{"task_slug":"open-vocabulary-object-detection","task_name":"Open Vocabulary Object Detection"},{"task_slug":"open-vocabulary-object-detection","task_name":"Open-vocabulary object detection"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2404.03539","atlas_url":"https://app.syntology.ai/?focus=2404.03539","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.03539"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lorebianchi98/fg-ovd","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lorebianchi98/fg-clip","reach":{"status":"ok"}}],"summary":{"ran":12,"ran_draft_wrong":3,"unverified":2},"by_repo_kind":{"official":{"samples":15,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":17,"samples":[{"code_sha256_prefix":"b36a8c39e91d5a90","entry":"clip_preds","repo":"lorebianchi98/fg-clip","repo_kind":"official","path":"src/ranks.py","file_url":"https://github.com/lorebianchi98/fg-clip/blob/HEAD/src/ranks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b36a8c39e91d5a90"}},{"code_sha256_prefix":"33d935018f3c899c","entry":"dict2pandas","repo":"lorebianchi98/fg-clip","repo_kind":"official","path":"src/eval_util.py","file_url":"https://github.com/lorebianchi98/fg-clip/blob/HEAD/src/eval_util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"33d935018f3c899c"}},{"code_sha256_prefix":"04513e995209d3a4","entry":"get_image_and_text_tensor","repo":"lorebianchi98/fg-clip","repo_kind":"official","path":"src/metrics.py","file_url":"https://github.com/lorebianchi98/fg-clip/blob/HEAD/src/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"04513e995209d3a4"}},{"code_sha256_prefix":"c18b5b3ed8247b90","entry":"get_karpathy_splits","repo":"lorebianchi98/fg-clip","repo_kind":"official","path":"features/create_karpathy_splits.py","file_url":"https://github.com/lorebianchi98/fg-clip/blob/HEAD/features/create_karpathy_splits.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c18b5b3ed8247b90"}},{"code_sha256_prefix":"8709e2ae58e458e2","entry":"get_ranks","repo":"lorebianchi98/fg-clip","repo_kind":"official","path":"src/eval_util.py","file_url":"https://github.com/lorebianchi98/fg-clip/blob/HEAD/src/eval_util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8709e2ae58e458e2"}},{"code_sha256_prefix":"2e9fc184fff19cb1","entry":"get_split_featured","repo":"lorebianchi98/fg-clip","repo_kind":"official","path":"features/create_karpathy_splits.py","file_url":"https://github.com/lorebianchi98/fg-clip/blob/HEAD/features/create_karpathy_splits.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2e9fc184fff19cb1"}},{"code_sha256_prefix":"7fe89e8b613a6f08","entry":"i2t","repo":"lorebianchi98/fg-clip","repo_kind":"official","path":"src/metrics.py","file_url":"https://github.com/lorebianchi98/fg-clip/blob/HEAD/src/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7fe89e8b613a6f08"}},{"code_sha256_prefix":"46323f57e9610d8e","entry":"load_object","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"46323f57e9610d8e"}},{"code_sha256_prefix":"124a87be7c2ae97b","entry":"out2coco_format","repo":"lorebianchi98/fg-clip","repo_kind":"official","path":"features/coco_feature_extractor.py","file_url":"https://github.com/lorebianchi98/fg-clip/blob/HEAD/features/coco_feature_extractor.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"124a87be7c2ae97b"}},{"code_sha256_prefix":"0be4cd21574acdc0","entry":"read_json","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"0be4cd21574acdc0"}},{"code_sha256_prefix":"5840bed4f36dd3fb","entry":"read_json","repo":"lorebianchi98/fg-clip","repo_kind":"official","path":"src/utilities.py","file_url":"https://github.com/lorebianchi98/fg-clip/blob/HEAD/src/utilities.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5840bed4f36dd3fb"}},{"code_sha256_prefix":"ce321b14ae1d05be","entry":"t2i","repo":"lorebianchi98/fg-clip","repo_kind":"official","path":"src/metrics.py","file_url":"https://github.com/lorebianchi98/fg-clip/blob/HEAD/src/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ce321b14ae1d05be"}},{"code_sha256_prefix":"cbeb45030a21b392","entry":"train","repo":"lorebianchi98/fg-clip","repo_kind":"official","path":"src/train_util.py","file_url":"https://github.com/lorebianchi98/fg-clip/blob/HEAD/src/train_util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cbeb45030a21b392"}},{"code_sha256_prefix":"876830dc323b0fa9","entry":"trainBCE","repo":"lorebianchi98/fg-clip","repo_kind":"official","path":"src/train_util.py","file_url":"https://github.com/lorebianchi98/fg-clip/blob/HEAD/src/train_util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"876830dc323b0fa9"}},{"code_sha256_prefix":"f702268bf37b191b","entry":"transform_predslist_to_dict","repo":"lorebianchi98/fg-clip","repo_kind":"official","path":"src/utilities.py","file_url":"https://github.com/lorebianchi98/fg-clip/blob/HEAD/src/utilities.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f702268bf37b191b"}},{"code_sha256_prefix":"9b0d3cf7536a3b55","entry":"choice_negatives","repo":"lorebianchi98/fg-clip","repo_kind":"official","path":"src/train_util.py","file_url":"https://github.com/lorebianchi98/fg-clip/blob/HEAD/src/train_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b0d3cf7536a3b55"}},{"code_sha256_prefix":"811f554b6375d712","entry":"load_object","repo":"lorebianchi98/fg-clip","repo_kind":"official","path":"src/utilities.py","file_url":"https://github.com/lorebianchi98/fg-clip/blob/HEAD/src/utilities.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"811f554b6375d712"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}