{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/from-text-to-mask-localizing-entities-using","title":"From Text to Mask: Localizing Entities Using the Attention of Text-to-Image Diffusion Models","arxiv_id":"2309.04109","date":"2023-09-08","proceeding":null,"authors":["Changming Xiao","Qi Yang","Feng Zhou","ChangShui Zhang"],"abstract":"Diffusion models have revolted the field of text-to-image generation recently. The unique way of fusing text and image information contributes to their remarkable capability of generating highly text-related images. From another perspective, these generative models imply clues about the precise correlation between words and pixels. In this work, a simple but effective method is proposed to utilize the attention mechanism in the denoising network of text-to-image diffusion models. Without re-training nor inference-time optimization, the semantic grounding of phrases can be attained directly. We evaluate our method on Pascal VOC 2012 and Microsoft COCO 2014 under weakly-supervised semantic segmentation setting and our method achieves superior performance to prior methods. In addition, the acquired word-pixel correlation is found to be generalizable for the learned text embedding of customized generation methods, requiring only a few modifications. To validate our discovery, we introduce a new practical task called \"personalized referring image segmentation\" with a new dataset. Experiments in various situations demonstrate the advantages of our method compared to strong baselines on this task. In summary, our work reveals a novel way to extract the rich multi-modal knowledge hidden in diffusion models for segmentation.","url_abs":"https://arxiv.org/abs/2309.04109v2","url_pdf":"https://arxiv.org/pdf/2309.04109v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"from-text-to-mask-localizing-entities-using","repo_url":"https://github.com/Big-Brother-Pikachu/Text2Mask","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"image-segmentation","task_name":"Image Segmentation"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"text-to-image-generation-1","task_name":"Text to Image Generation"},{"task_slug":"text-to-image-generation","task_name":"Text-to-Image Generation"},{"task_slug":"weakly-supervised-semantic-segmentation-1","task_name":"Weakly supervised Semantic Segmentation"},{"task_slug":"weakly-supervised-semantic-segmentation","task_name":"Weakly-Supervised Semantic Segmentation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"latent-diffusion-model","method_name":"Latent Diffusion Model"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/weakly-supervised-semantic-segmentation-on-4","task":"Weakly-Supervised Semantic Segmentation","dataset":"COCO 2014 val","model":"T2MDiffusion(DeepLabV2-ResNet101)","rank_in_archive_order":11,"of":39,"metrics":{"mIoU":"45.7"},"uses_additional_data":true},{"leaderboard":"/sota/weakly-supervised-semantic-segmentation-on-1","task":"Weakly-Supervised Semantic Segmentation","dataset":"PASCAL VOC 2012 test","model":"T2MDiffusion(DeepLabV2-ResNet101)","rank_in_archive_order":11,"of":60,"metrics":{"Mean IoU":"74.2"},"uses_additional_data":true},{"leaderboard":"/sota/weakly-supervised-semantic-segmentation-on","task":"Weakly-Supervised Semantic Segmentation","dataset":"PASCAL VOC 2012 val","model":"T2MDiffusion(DeepLabV2-ResNet101)","rank_in_archive_order":16,"of":73,"metrics":{"Mean IoU":"73.3"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.04109","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}