{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/groupvit-semantic-segmentation-emerges-from","title":"GroupViT: Semantic Segmentation Emerges from Text Supervision","arxiv_id":"2202.11094","date":"2022-02-22","proceeding":"CVPR 2022 1","authors":["Jiarui Xu","Shalini De Mello","Sifei Liu","Wonmin Byeon","Thomas Breuel","Jan Kautz","Xiaolong Wang"],"abstract":"Grouping and recognition are important components of visual scene understanding, e.g., for object detection and semantic segmentation. With end-to-end deep learning systems, grouping of image regions usually happens implicitly via top-down supervision from pixel-level recognition labels. Instead, in this paper, we propose to bring back the grouping mechanism into deep networks, which allows semantic segments to emerge automatically with only text supervision. We propose a hierarchical Grouping Vision Transformer (GroupViT), which goes beyond the regular grid structure representation and learns to group image regions into progressively larger arbitrary-shaped segments. We train GroupViT jointly with a text encoder on a large-scale image-text dataset via contrastive losses. With only text supervision and without any pixel-level annotations, GroupViT learns to group together semantic regions and successfully transfers to the task of semantic segmentation in a zero-shot manner, i.e., without any further fine-tuning. It achieves a zero-shot accuracy of 52.3% mIoU on the PASCAL VOC 2012 and 22.4% mIoU on PASCAL Context datasets, and performs competitively to state-of-the-art transfer-learning methods requiring greater levels of supervision. We open-source our code at https://github.com/NVlabs/GroupViT .","url_abs":"https://arxiv.org/abs/2202.11094v5","url_pdf":"https://arxiv.org/pdf/2202.11094v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"groupvit-semantic-segmentation-emerges-from","repo_url":"https://github.com/NVlabs/GroupViT","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"groupvit-semantic-segmentation-emerges-from","repo_url":"https://github.com/huggingface/transformers","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"groupvit-semantic-segmentation-emerges-from","repo_url":"https://github.com/2024-MindSpore-1/Code2/tree/main/model-1/groupvit","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"groupvit-semantic-segmentation-emerges-from","repo_url":"https://github.com/MindSpore-scientific-2/code-14/tree/main/groupvit","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"groupvit-semantic-segmentation-emerges-from","repo_url":"https://github.com/pwc-1/Paper-9/tree/main/groupvit","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"groupvit-semantic-segmentation-emerges-from","repo_url":"https://github.com/yangyucheng000/University/tree/main/model-2/groupvit","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"scene-understanding","task_name":"Scene Understanding"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"},{"task_slug":"unsupervised-semantic-segmentation-with","task_name":"Unsupervised Semantic Segmentation with Language-image Pre-training"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/unsupervised-semantic-segmentation-with-4","task":"Unsupervised Semantic Segmentation with Language-image Pre-training","dataset":"ADE20K","model":"GroupViT (RedCaps)","rank_in_archive_order":13,"of":13,"metrics":{"Mean IoU (val)":"9.2"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-semantic-segmentation-with-10","task":"Unsupervised Semantic Segmentation with Language-image Pre-training","dataset":"COCO-Object","model":"GroupViT (RedCaps)","rank_in_archive_order":9,"of":12,"metrics":{"mIoU":"27.5"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-semantic-segmentation-with-9","task":"Unsupervised Semantic Segmentation with Language-image Pre-training","dataset":"COCO-Stuff-171","model":"GroupViT","rank_in_archive_order":12,"of":12,"metrics":{"mIoU":"11.1"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-semantic-segmentation-with-8","task":"Unsupervised Semantic Segmentation with Language-image Pre-training","dataset":"PASCAL Context-59","model":"GroupViT (RedCaps)","rank_in_archive_order":11,"of":12,"metrics":{"mIoU":"23.4"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-semantic-segmentation-with-7","task":"Unsupervised Semantic Segmentation with Language-image Pre-training","dataset":"PascalVOC-20","model":"GroupViT (RedCaps)","rank_in_archive_order":7,"of":10,"metrics":{"mIoU":"79.7"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2202.11094","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}