{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/detailclip-detail-oriented-clip-for-fine","title":"DetailCLIP: Detail-Oriented CLIP for Fine-Grained Tasks","arxiv_id":"2409.06809","date":"2024-09-10","proceeding":null,"authors":["Amin Karimi Monsefi","Kishore Prakash Sailaja","Ali Alilooee","Ser-Nam Lim","Rajiv Ramnath"],"abstract":"In this paper, we introduce DetailCLIP: A Detail-Oriented CLIP to address the limitations of contrastive learning-based vision-language models, particularly CLIP, in handling detail-oriented and fine-grained tasks like segmentation. While CLIP and its variants excel in the global alignment of image and text representations, they often struggle to capture the fine-grained details necessary for precise segmentation. To overcome these challenges, we propose a novel framework that employs patch-level comparison of self-distillation and pixel-level reconstruction losses, enhanced with an attention-based token removal mechanism. This approach selectively retains semantically relevant tokens, enabling the model to focus on the image's critical regions aligned with the specific functions of our model, including textual information processing, patch comparison, and image reconstruction, ensuring that the model learns high-level semantics and detailed visual features. Our experiments demonstrate that DetailCLIP surpasses existing CLIP-based and traditional self-supervised learning (SSL) models in segmentation accuracy and exhibits superior generalization across diverse datasets. DetailCLIP represents a significant advancement in vision-language modeling, offering a robust solution for tasks that demand high-level semantic understanding and detailed feature extraction. https://github.com/KishoreP1/DetailCLIP.","url_abs":"https://arxiv.org/abs/2409.06809v2","url_pdf":"https://arxiv.org/pdf/2409.06809v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"detailclip-detail-oriented-clip-for-fine","repo_url":"https://github.com/KishoreP1/DetailCLIP","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"image-reconstruction","task_name":"Image Reconstruction"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.06809","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.06809"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/KishoreP1/DetailCLIP","reach":null}],"summary":{"ran":1,"ran_fixture":3,"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"official":{"samples":11,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"8d7b5685c27c8566","entry":"CLIP","repo":"KishoreP1/DetailCLIP","repo_kind":"official","path":"models.py","file_url":"https://github.com/KishoreP1/DetailCLIP/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8d7b5685c27c8566"}},{"code_sha256_prefix":"2e59707d16d21473","entry":"accuracy","repo":"KishoreP1/DetailCLIP","repo_kind":"official","path":"eval_zeroshot.py","file_url":"https://github.com/KishoreP1/DetailCLIP/blob/HEAD/eval_zeroshot.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2e59707d16d21473"}},{"code_sha256_prefix":"4f14f3e2415a28c0","entry":"extract","repo":"KishoreP1/DetailCLIP","repo_kind":"official","path":"extract_backbone_weights.py","file_url":"https://github.com/KishoreP1/DetailCLIP/blob/HEAD/extract_backbone_weights.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4f14f3e2415a28c0"}},{"code_sha256_prefix":"f8278dbfe57c1b58","entry":"get_att_mask","repo":"KishoreP1/DetailCLIP","repo_kind":"official","path":"models.py","file_url":"https://github.com/KishoreP1/DetailCLIP/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f8278dbfe57c1b58"}},{"code_sha256_prefix":"f0eb8193f86d363b","entry":"mean_per_class","repo":"KishoreP1/DetailCLIP","repo_kind":"official","path":"eval_zeroshot.py","file_url":"https://github.com/KishoreP1/DetailCLIP/blob/HEAD/eval_zeroshot.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f0eb8193f86d363b"}},{"code_sha256_prefix":"c33ed4816cfeb73f","entry":"CSyncBatchNorm","repo":"KishoreP1/DetailCLIP","repo_kind":"official","path":"models.py","file_url":"https://github.com/KishoreP1/DetailCLIP/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c33ed4816cfeb73f"}},{"code_sha256_prefix":"39a3b3f9967fdec3","entry":"CustomSequential","repo":"KishoreP1/DetailCLIP","repo_kind":"official","path":"models.py","file_url":"https://github.com/KishoreP1/DetailCLIP/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39a3b3f9967fdec3"}},{"code_sha256_prefix":"1fe58b925a6046d7","entry":"DINOHead","repo":"KishoreP1/DetailCLIP","repo_kind":"official","path":"models.py","file_url":"https://github.com/KishoreP1/DetailCLIP/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1fe58b925a6046d7"}},{"code_sha256_prefix":"857796c6313780ab","entry":"DetailCLIP","repo":"KishoreP1/DetailCLIP","repo_kind":"official","path":"models.py","file_url":"https://github.com/KishoreP1/DetailCLIP/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"857796c6313780ab"}},{"code_sha256_prefix":"90448ee89aa26b11","entry":"PSyncBatchNorm","repo":"KishoreP1/DetailCLIP","repo_kind":"official","path":"models.py","file_url":"https://github.com/KishoreP1/DetailCLIP/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"90448ee89aa26b11"}},{"code_sha256_prefix":"16d539a3bde9714a","entry":"iBOTHead","repo":"KishoreP1/DetailCLIP","repo_kind":"official","path":"models.py","file_url":"https://github.com/KishoreP1/DetailCLIP/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"16d539a3bde9714a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}