{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/open-vocabulary-semantic-segmentation-with","title":"Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP","arxiv_id":"2210.04150","date":"2022-10-09","proceeding":"CVPR 2023 1","authors":["Feng Liang","Bichen Wu","Xiaoliang Dai","Kunpeng Li","Yinan Zhao","Hang Zhang","Peizhao Zhang","Peter Vajda","Diana Marculescu"],"abstract":"Open-vocabulary semantic segmentation aims to segment an image into semantic regions according to text descriptions, which may not have been seen during training. Recent two-stage methods first generate class-agnostic mask proposals and then leverage pre-trained vision-language models, e.g., CLIP, to classify masked regions. We identify the performance bottleneck of this paradigm to be the pre-trained CLIP model, since it does not perform well on masked images. To address this, we propose to finetune CLIP on a collection of masked image regions and their corresponding text descriptions. We collect training data by mining an existing image-caption dataset (e.g., COCO Captions), using CLIP to match masked image regions to nouns in the image captions. Compared with the more precise and manually annotated segmentation labels with fixed classes (e.g., COCO-Stuff), we find our noisy but diverse dataset can better retain CLIP's generalization ability. Along with finetuning the entire model, we utilize the \"blank\" areas in masked images using a method we dub mask prompt tuning. Experiments demonstrate mask prompt tuning brings significant improvement without modifying any weights of CLIP, and it can further improve a fully finetuned model. In particular, when trained on COCO and evaluated on ADE20K-150, our best model achieves 29.6% mIoU, which is +8.5% higher than the previous state-of-the-art. For the first time, open-vocabulary generalist models match the performance of supervised specialist models in 2017 without dataset-specific adaptations.","url_abs":"https://arxiv.org/abs/2210.04150v3","url_pdf":"https://arxiv.org/pdf/2210.04150v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"open-vocabulary-semantic-segmentation-with","repo_url":"https://github.com/facebookresearch/ov-seg","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"open-vocabulary-semantic-segmentation","task_name":"Open Vocabulary Semantic Segmentation"},{"task_slug":"open-vocabulary-semantic-segmentation-1","task_name":"Open-Vocabulary Semantic Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/open-vocabulary-semantic-segmentation-on-2","task":"Open Vocabulary Semantic Segmentation","dataset":"ADE20K-150","model":"OVSeg Swin-B","rank_in_archive_order":17,"of":23,"metrics":{"mIoU":"29.6"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-semantic-segmentation-on-3","task":"Open Vocabulary Semantic Segmentation","dataset":"ADE20K-847","model":"OVSeg Swin-B","rank_in_archive_order":17,"of":19,"metrics":{"mIoU":"9"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-semantic-segmentation-on-7","task":"Open Vocabulary Semantic Segmentation","dataset":"PASCAL Context-459","model":"OVSeg Swin-B","rank_in_archive_order":14,"of":15,"metrics":{"mIoU":"12.4"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-semantic-segmentation-on-1","task":"Open Vocabulary Semantic Segmentation","dataset":"PASCAL Context-59","model":"OVSeg Swin-B","rank_in_archive_order":15,"of":24,"metrics":{"mIoU":"55.7"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-semantic-segmentation-on-5","task":"Open Vocabulary Semantic Segmentation","dataset":"PascalVOC-20","model":"OVSeg Swin-B","rank_in_archive_order":9,"of":20,"metrics":{"mIoU":"94.5"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-replica","task":"Semantic Segmentation","dataset":"Replica","model":"OVSeg","rank_in_archive_order":4,"of":5,"metrics":{"mIoU":"20.7"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.04150","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.04150"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/ov-seg","reach":null}],"summary":{"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"8e95157106d1df8c","entry":"trace_model","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"8e95157106d1df8c"}},{"code_sha256_prefix":"b9b4b4cc71fba508","entry":"build_model_from_openai_state_dict","repo":"facebookresearch/ov-seg","repo_kind":"official","path":"open_clip_training/src/open_clip/model.py","file_url":"https://github.com/facebookresearch/ov-seg/blob/HEAD/open_clip_training/src/open_clip/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b9b4b4cc71fba508"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}