{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/corrclip-reconstructing-correlations-in-clip","title":"CorrCLIP: Reconstructing Correlations in CLIP with Off-the-Shelf Foundation Models for Open-Vocabulary Semantic Segmentation","arxiv_id":"2411.10086","date":"2024-11-15","proceeding":null,"authors":["Dengke Zhang","Fagui Liu","Quan Tang"],"abstract":"Open-vocabulary semantic segmentation aims to assign semantic labels to each pixel without relying on a predefined set of categories. Contrastive Language-Image Pre-training (CLIP) demonstrates outstanding zero-shot classification capabilities but struggles with the pixel-wise segmentation task as the captured inter-patch correlations correspond to no specific visual concepts. Despite previous CLIP-based works improving inter-patch correlations by self-self attention, they still face the inherent limitation that image patches tend to have high similarity to outlier ones. In this work, we introduce CorrCLIP, a training-free approach for open-vocabulary semantic segmentation, which reconstructs significantly coherent inter-patch correlations utilizing foundation models. Specifically, it employs the Segment Anything Model (SAM) to define the scope of patch interactions, ensuring that patches interact only with semantically similar ones. Furthermore, CorrCLIP obtains an understanding of an image's semantic layout via self-supervised models to determine concrete similarity values between image patches, which addresses the similarity irregularity problem caused by the aforementioned restricted patch interaction regime. Finally, CorrCLIP reuses the region masks produced by SAM to update the segmentation map. As a training-free method, CorrCLIP achieves a notable improvement across eight challenging benchmarks regarding the averaged mean Intersection over Union, boosting it from 44.4% to 51.0%.","url_abs":"https://arxiv.org/abs/2411.10086v1","url_pdf":"https://arxiv.org/pdf/2411.10086v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"corrclip-reconstructing-correlations-in-clip","repo_url":"https://github.com/zdk258/CorrCLIP","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"open-vocabulary-semantic-segmentation","task_name":"Open Vocabulary Semantic Segmentation"},{"task_slug":"open-vocabulary-semantic-segmentation-1","task_name":"Open-Vocabulary Semantic Segmentation"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"unsupervised-semantic-segmentation-with","task_name":"Unsupervised Semantic Segmentation with Language-image Pre-training"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"},{"task_slug":null,"task_name":"zero-shot-classification"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"dino","method_name":"DINO"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"sam","method_name":"SAM"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/unsupervised-semantic-segmentation-with-4","task":"Unsupervised Semantic Segmentation with Language-image Pre-training","dataset":"ADE20K","model":"CorrCLIP","rank_in_archive_order":1,"of":13,"metrics":{"Mean IoU (val)":"30.7"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-semantic-segmentation-with-10","task":"Unsupervised Semantic Segmentation with Language-image Pre-training","dataset":"COCO-Object","model":"CorrCLIP","rank_in_archive_order":1,"of":12,"metrics":{"mIoU":"49.4"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-semantic-segmentation-with-9","task":"Unsupervised Semantic Segmentation with Language-image Pre-training","dataset":"COCO-Stuff-171","model":"CorrCLIP","rank_in_archive_order":1,"of":12,"metrics":{"mIoU":"34.0"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-semantic-segmentation-with-3","task":"Unsupervised Semantic Segmentation with Language-image Pre-training","dataset":"Cityscapes val","model":"CorrCLIP","rank_in_archive_order":1,"of":12,"metrics":{"mIoU":"51.1"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-semantic-segmentation-with-8","task":"Unsupervised Semantic Segmentation with Language-image Pre-training","dataset":"PASCAL Context-59","model":"CorrCLIP","rank_in_archive_order":1,"of":12,"metrics":{"mIoU":"50.8"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-semantic-segmentation-with-12","task":"Unsupervised Semantic Segmentation with Language-image Pre-training","dataset":"PASCAL Context-60","model":"CorrCLIP","rank_in_archive_order":1,"of":4,"metrics":{"mIoU":"44.9"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-semantic-segmentation-with-11","task":"Unsupervised Semantic Segmentation with Language-image Pre-training","dataset":"PASCAL VOC","model":"CorrCLIP","rank_in_archive_order":1,"of":10,"metrics":{"mIoU":"76.7"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-semantic-segmentation-with-7","task":"Unsupervised Semantic Segmentation with Language-image Pre-training","dataset":"PascalVOC-20","model":"CorrCLIP","rank_in_archive_order":1,"of":10,"metrics":{"mIoU":"91.8"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.10086","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.10086"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zdk258/CorrCLIP","reach":{"status":"ok"}}],"summary":{"unverified":9},"by_repo_kind":{"official":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"ddcbd45e940484ee","entry":"gather_features","repo":"zdk258/CorrCLIP","repo_kind":"official","path":"CorrCLIPv1/open_clip/loss.py","file_url":"https://github.com/zdk258/CorrCLIP/blob/HEAD/CorrCLIPv1/open_clip/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ddcbd45e940484ee"}},{"code_sha256_prefix":"38fc34495256c6cf","entry":"generate_distinct_colors","repo":"zdk258/CorrCLIP","repo_kind":"official","path":"CorrCLIPv1/demo_colab.py","file_url":"https://github.com/zdk258/CorrCLIP/blob/HEAD/CorrCLIPv1/demo_colab.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"38fc34495256c6cf"}},{"code_sha256_prefix":"8e6ee615153e5dfb","entry":"generate_distinct_colors","repo":"zdk258/CorrCLIP","repo_kind":"official","path":"CorrCLIPv1/demo_gradio.py","file_url":"https://github.com/zdk258/CorrCLIP/blob/HEAD/CorrCLIPv1/demo_gradio.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8e6ee615153e5dfb"}},{"code_sha256_prefix":"54731d24b3557840","entry":"get_cls_idx","repo":"zdk258/CorrCLIP","repo_kind":"official","path":"CorrCLIPv1/corrclip_segmentor.py","file_url":"https://github.com/zdk258/CorrCLIP/blob/HEAD/CorrCLIPv1/corrclip_segmentor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"54731d24b3557840"}},{"code_sha256_prefix":"e332856e3c2fc814","entry":"neighbour_exchange","repo":"zdk258/CorrCLIP","repo_kind":"official","path":"CorrCLIPv1/open_clip/loss.py","file_url":"https://github.com/zdk258/CorrCLIP/blob/HEAD/CorrCLIPv1/open_clip/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e332856e3c2fc814"}},{"code_sha256_prefix":"5b1fd364afcf3c05","entry":"neighbour_exchange_bidir","repo":"zdk258/CorrCLIP","repo_kind":"official","path":"CorrCLIPv1/open_clip/loss.py","file_url":"https://github.com/zdk258/CorrCLIP/blob/HEAD/CorrCLIPv1/open_clip/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5b1fd364afcf3c05"}},{"code_sha256_prefix":"fb651d0a97fd4d3f","entry":"prepare_inputs_for_generation","repo":"zdk258/CorrCLIP","repo_kind":"official","path":"CorrCLIPv1/open_clip/coca_model.py","file_url":"https://github.com/zdk258/CorrCLIP/blob/HEAD/CorrCLIPv1/open_clip/coca_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fb651d0a97fd4d3f"}},{"code_sha256_prefix":"2a377da4a76a2d44","entry":"register_pooler","repo":"zdk258/CorrCLIP","repo_kind":"official","path":"CorrCLIPv1/open_clip/hf_model.py","file_url":"https://github.com/zdk258/CorrCLIP/blob/HEAD/CorrCLIPv1/open_clip/hf_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2a377da4a76a2d44"}},{"code_sha256_prefix":"434ff5d8530045d3","entry":"run_segmentation","repo":"zdk258/CorrCLIP","repo_kind":"official","path":"CorrCLIPv1/demo_colab.py","file_url":"https://github.com/zdk258/CorrCLIP/blob/HEAD/CorrCLIPv1/demo_colab.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"434ff5d8530045d3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}