{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/uncovering-prototypical-knowledge-for-weakly","title":"Uncovering Prototypical Knowledge for Weakly Open-Vocabulary Semantic Segmentation","arxiv_id":"2310.19001","date":"2023-10-29","proceeding":"NeurIPS 2023 11","authors":[],"abstract":"This paper studies the problem of weakly open-vocabulary semantic segmentation (WOVSS), which learns to segment objects of arbitrary classes using mere image-text pairs. Existing works turn to enhance the vanilla vision transformer by introducing explicit grouping recognition, i.e., employing several group tokens/centroids to cluster the image tokens and perform the group-text alignment. Nevertheless, these methods suffer from a granularity inconsistency regarding the usage of group tokens, which are aligned in the all-to-one v.s. one-to-one manners during the training and inference phases, respectively. We argue that this discrepancy arises from the lack of elaborate supervision for each group token. To bridge this granularity gap, this paper explores explicit supervision for the group tokens from the prototypical knowledge. To this end, this paper proposes the non-learnable prototypical regularization (NPR) where non-learnable prototypes are estimated from source features to serve as supervision and enable contrastive matching of the group tokens. This regularization encourages the group tokens to segment objects with less redundancy and capture more comprehensive semantic regions, leading to increased compactness and richness. Based on NPR, we propose the prototypical guidance segmentation network (PGSeg) that incorporates multi-modal regularization by leveraging prototypical sources from both images and texts at different levels, progressively enhancing the segmentation capability with diverse prototypical patterns. Experimental results show that our proposed method achieves state-of-the-art performance on several benchmark datasets. The source code is available at https://github.com/Ferenas/PGSeg.","url_abs":"https://arxiv.org/abs/2310.19001v1","url_pdf":"https://arxiv.org/pdf/2310.19001v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"uncovering-prototypical-knowledge-for-weakly","repo_url":"https://github.com/ferenas/pgseg","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"open-vocabulary-semantic-segmentation","task_name":"Open Vocabulary Semantic Segmentation"},{"task_slug":"open-vocabulary-semantic-segmentation-1","task_name":"Open-Vocabulary Semantic Segmentation"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.19001","atlas_url":"https://app.syntology.ai/?focus=2310.19001","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.19001"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Ferenas/PGSeg","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ferenas/pgseg","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":2,"ran":3,"unverified":1},"by_repo_kind":{"listed":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"7f29142583965575","entry":"gumbel_softmax","repo":"Ferenas/PGSeg","repo_kind":"listed","path":"models/pgseg.py","file_url":"https://github.com/Ferenas/PGSeg/blob/HEAD/models/pgseg.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7f29142583965575"}},{"code_sha256_prefix":"cb0acca1ea9ead1e","entry":"hard_softmax","repo":"Ferenas/PGSeg","repo_kind":"listed","path":"models/pgseg.py","file_url":"https://github.com/Ferenas/PGSeg/blob/HEAD/models/pgseg.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb0acca1ea9ead1e"}},{"code_sha256_prefix":"af0481096573c066","entry":"interpolate_pos_encoding","repo":"Ferenas/PGSeg","repo_kind":"listed","path":"models/misc.py","file_url":"https://github.com/Ferenas/PGSeg/blob/HEAD/models/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"af0481096573c066"}},{"code_sha256_prefix":"0fe0836e8952564b","entry":"seg2coord","repo":"Ferenas/PGSeg","repo_kind":"listed","path":"segmentation/evaluation/pgseg_seg.py","file_url":"https://github.com/Ferenas/PGSeg/blob/HEAD/segmentation/evaluation/pgseg_seg.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0fe0836e8952564b"}},{"code_sha256_prefix":"3cd1eee26ae01a18","entry":"top_groups","repo":"Ferenas/PGSeg","repo_kind":"listed","path":"segmentation/evaluation/pgseg_seg.py","file_url":"https://github.com/Ferenas/PGSeg/blob/HEAD/segmentation/evaluation/pgseg_seg.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3cd1eee26ae01a18"}},{"code_sha256_prefix":"be236b35deddb211","entry":"resize_attn_map","repo":"Ferenas/PGSeg","repo_kind":"listed","path":"segmentation/evaluation/pgseg_seg.py","file_url":"https://github.com/Ferenas/PGSeg/blob/HEAD/segmentation/evaluation/pgseg_seg.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"be236b35deddb211"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}