{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-mask-aware-clip-representations-for","title":"Learning Mask-aware CLIP Representations for Zero-Shot Segmentation","arxiv_id":"2310.00240","date":"2023-09-30","proceeding":"NeurIPS 2023 11","authors":["Siyu Jiao","Yunchao Wei","YaoWei Wang","Yao Zhao","Humphrey Shi"],"abstract":"Recently, pre-trained vision-language models have been increasingly used to tackle the challenging zero-shot segmentation task. Typical solutions follow the paradigm of first generating mask proposals and then adopting CLIP to classify them. To maintain the CLIP's zero-shot transferability, previous practices favour to freeze CLIP during training. However, in the paper, we reveal that CLIP is insensitive to different mask proposals and tends to produce similar predictions for various mask proposals of the same image. This insensitivity results in numerous false positives when classifying mask proposals. This issue mainly relates to the fact that CLIP is trained with image-level supervision. To alleviate this issue, we propose a simple yet effective method, named Mask-aware Fine-tuning (MAFT). Specifically, Image-Proposals CLIP Encoder (IP-CLIP Encoder) is proposed to handle arbitrary numbers of image and mask proposals simultaneously. Then, mask-aware loss and self-distillation loss are designed to fine-tune IP-CLIP Encoder, ensuring CLIP is responsive to different mask proposals while not sacrificing transferability. In this way, mask-aware representations can be easily learned to make the true positives stand out. Notably, our solution can seamlessly plug into most existing methods without introducing any new parameters during the fine-tuning process. We conduct extensive experiments on the popular zero-shot benchmarks. With MAFT, the performance of the state-of-the-art methods is promoted by a large margin: 50.4% (+ 8.2%) on COCO, 81.8% (+ 3.2%) on Pascal-VOC, and 8.7% (+4.3%) on ADE20K in terms of mIoU for unseen classes. The code is available at https://github.com/jiaosiyu1999/MAFT.git.","url_abs":"https://arxiv.org/abs/2310.00240v1","url_pdf":"https://arxiv.org/pdf/2310.00240v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-mask-aware-clip-representations-for","repo_url":"https://github.com/jiaosiyu1999/maft","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"learning-mask-aware-clip-representations-for","repo_url":"https://github.com/jiaosiyu1999/MAFT-Plus","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"open-vocabulary-semantic-segmentation","task_name":"Open Vocabulary Semantic Segmentation"},{"task_slug":"zero-shot-segmentation","task_name":"Zero Shot Segmentation"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/open-vocabulary-semantic-segmentation-on-2","task":"Open Vocabulary Semantic Segmentation","dataset":"ADE20K-150","model":"MAFT-ViTL","rank_in_archive_order":14,"of":23,"metrics":{"mIoU":"32.0"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-semantic-segmentation-on-3","task":"Open Vocabulary Semantic Segmentation","dataset":"ADE20K-847","model":"MAFT-ViTL","rank_in_archive_order":15,"of":19,"metrics":{"mIoU":"12.1"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-semantic-segmentation-on-7","task":"Open Vocabulary Semantic Segmentation","dataset":"PASCAL Context-459","model":"MAFT-ViTL","rank_in_archive_order":12,"of":15,"metrics":{"mIoU":"15.7"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-semantic-segmentation-on-1","task":"Open Vocabulary Semantic Segmentation","dataset":"PASCAL Context-59","model":"MAFT-ViTL","rank_in_archive_order":12,"of":24,"metrics":{"mIoU":"58.5"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-semantic-segmentation-on-5","task":"Open Vocabulary Semantic Segmentation","dataset":"PascalVOC-20","model":"MAFT-ViTL","rank_in_archive_order":11,"of":20,"metrics":{"mIoU":"92.1"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.00240","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.00240"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/jiaosiyu1999/MAFT","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jiaosiyu1999/MAFT-Plus","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jiaosiyu1999/maft","reach":{"status":"ok"}}],"summary":{"ran":1,"unverified":6},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1},"listed":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"6935590148878098","entry":"loadAde20K","repo":"jiaosiyu1999/MAFT","repo_kind":"official","path":"datasets/prepare_ade20k_full_sem_seg.py","file_url":"https://github.com/jiaosiyu1999/MAFT/blob/HEAD/datasets/prepare_ade20k_full_sem_seg.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6935590148878098"}},{"code_sha256_prefix":"bc2cb481a75c370d","entry":"batch_dice_loss","repo":"jiaosiyu1999/MAFT-Plus","repo_kind":"listed","path":"maft/modeling/matcher.py","file_url":"https://github.com/jiaosiyu1999/MAFT-Plus/blob/HEAD/maft/modeling/matcher.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc2cb481a75c370d"}},{"code_sha256_prefix":"1edd24985036b0bf","entry":"batch_sigmoid_ce_loss","repo":"jiaosiyu1999/MAFT-Plus","repo_kind":"listed","path":"maft/modeling/matcher.py","file_url":"https://github.com/jiaosiyu1999/MAFT-Plus/blob/HEAD/maft/modeling/matcher.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1edd24985036b0bf"}},{"code_sha256_prefix":"2dcb8123d89bb1ff","entry":"calculate_uncertainty","repo":"jiaosiyu1999/MAFT-Plus","repo_kind":"listed","path":"maft/modeling/criterion.py","file_url":"https://github.com/jiaosiyu1999/MAFT-Plus/blob/HEAD/maft/modeling/criterion.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2dcb8123d89bb1ff"}},{"code_sha256_prefix":"89f75e54ff128be0","entry":"dice_loss","repo":"jiaosiyu1999/MAFT-Plus","repo_kind":"listed","path":"maft/modeling/criterion.py","file_url":"https://github.com/jiaosiyu1999/MAFT-Plus/blob/HEAD/maft/modeling/criterion.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"89f75e54ff128be0"}},{"code_sha256_prefix":"cb6dcd02dbb95f73","entry":"generate_labels","repo":"jiaosiyu1999/MAFT","repo_kind":"official","path":"datasets/prepare_pascal_ctx_sem_seg.py","file_url":"https://github.com/jiaosiyu1999/MAFT/blob/HEAD/datasets/prepare_pascal_ctx_sem_seg.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb6dcd02dbb95f73"}},{"code_sha256_prefix":"d0c61e8dba511aa3","entry":"sigmoid_ce_loss","repo":"jiaosiyu1999/MAFT-Plus","repo_kind":"listed","path":"maft/modeling/criterion.py","file_url":"https://github.com/jiaosiyu1999/MAFT-Plus/blob/HEAD/maft/modeling/criterion.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d0c61e8dba511aa3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}