{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/open-vocabulary-instance-segmentation-via","title":"Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling","arxiv_id":"2111.12698","date":"2021-11-24","proceeding":"CVPR 2022 1","authors":["Dat Huynh","Jason Kuen","Zhe Lin","Jiuxiang Gu","Ehsan Elhamifar"],"abstract":"Open-vocabulary instance segmentation aims at segmenting novel classes without mask annotations. It is an important step toward reducing laborious human supervision. Most existing works first pretrain a model on captioned images covering many novel classes and then finetune it on limited base classes with mask annotations. However, the high-level textual information learned from caption pretraining alone cannot effectively encode the details required for pixel-wise segmentation. To address this, we propose a cross-modal pseudo-labeling framework, which generates training pseudo masks by aligning word semantics in captions with visual features of object masks in images. Thus, our framework is capable of labeling novel classes in captions via their word semantics to self-train a student model. To account for noises in pseudo masks, we design a robust student model that selectively distills mask knowledge by estimating the mask noise levels, hence mitigating the adverse impact of noisy pseudo masks. By extensive experiments, we show the effectiveness of our framework, where we significantly improve mAP score by 4.5% on MS-COCO and 5.1% on the large-scale Open Images & Conceptual Captions datasets compared to the state-of-the-art.","url_abs":"https://arxiv.org/abs/2111.12698v2","url_pdf":"https://arxiv.org/pdf/2111.12698v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"open-vocabulary-instance-segmentation-via","repo_url":"https://github.com/hbdat/cvpr22_cross_modal_pseudo_labeling","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2111.12698","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2111.12698"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hbdat/cvpr22_cross_modal_pseudo_labeling","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":2,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"c42eaa8eb1b704cd","entry":"sigmoid_focal_loss_cpu","repo":"hbdat/cvpr22_cross_modal_pseudo_labeling","repo_kind":"official","path":"maskrcnn_benchmark/layers/sigmoid_focal_loss.py","file_url":"https://github.com/hbdat/cvpr22_cross_modal_pseudo_labeling/blob/HEAD/maskrcnn_benchmark/layers/sigmoid_focal_loss.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"c42eaa8eb1b704cd"}},{"code_sha256_prefix":"e261fa29066b37e5","entry":"smooth_l1_loss","repo":"hbdat/cvpr22_cross_modal_pseudo_labeling","repo_kind":"official","path":"maskrcnn_benchmark/layers/smooth_l1_loss.py","file_url":"https://github.com/hbdat/cvpr22_cross_modal_pseudo_labeling/blob/HEAD/maskrcnn_benchmark/layers/smooth_l1_loss.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"e261fa29066b37e5"}},{"code_sha256_prefix":"83c64313d0501174","entry":"get_group_gn","repo":"hbdat/cvpr22_cross_modal_pseudo_labeling","repo_kind":"official","path":"maskrcnn_benchmark/modeling/make_layers.py","file_url":"https://github.com/hbdat/cvpr22_cross_modal_pseudo_labeling/blob/HEAD/maskrcnn_benchmark/modeling/make_layers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"83c64313d0501174"}},{"code_sha256_prefix":"2902bf4410253ff7","entry":"interpolate","repo":"hbdat/cvpr22_cross_modal_pseudo_labeling","repo_kind":"official","path":"maskrcnn_benchmark/layers/misc.py","file_url":"https://github.com/hbdat/cvpr22_cross_modal_pseudo_labeling/blob/HEAD/maskrcnn_benchmark/layers/misc.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"2902bf4410253ff7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}