{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tagalign-improving-vision-language-alignment","title":"TagAlign: Improving Vision-Language Alignment with Multi-Tag Classification","arxiv_id":"2312.14149","date":"2023-12-21","proceeding":null,"authors":["Qinying Liu","Wei Wu","Kecheng Zheng","Zhan Tong","Jiawei Liu","Yu Liu","Wei Chen","Zilei Wang","Yujun Shen"],"abstract":"The crux of learning vision-language models is to extract semantically aligned information from visual and linguistic data. Existing attempts usually face the problem of coarse alignment, e.g., the vision encoder struggles in localizing an attribute-specified object. In this work, we propose an embarrassingly simple approach to better align image and text features with no need of additional data formats other than image-text pairs. Concretely, given an image and its paired text, we manage to parse objects (e.g., cat) and attributes (e.g., black) from the description, which are highly likely to exist in the image. It is noteworthy that the parsing pipeline is fully automatic and thus enjoys good scalability. With these parsed semantics as supervision signals, we can complement the commonly used image-text contrastive loss with the multi-tag classification loss. Extensive experimental results on a broad suite of semantic segmentation datasets substantiate the average 5.2\\% improvement of our framework over existing alternatives. Furthermore, the visualization results indicate that attribute supervision makes vision-language models accurately localize attribute-specified objects. Project page can be found at https://qinying-liu.github.io/Tag-Align.","url_abs":"https://arxiv.org/abs/2312.14149v4","url_pdf":"https://arxiv.org/pdf/2312.14149v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tagalign-improving-vision-language-alignment","repo_url":"https://github.com/Qinying-Liu/TagAlign","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"open-vocabulary-semantic-segmentation","task_name":"Open Vocabulary Semantic Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"tag","task_name":"TAG"},{"task_slug":"unsupervised-semantic-segmentation-with","task_name":"Unsupervised Semantic Segmentation with Language-image Pre-training"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/open-vocabulary-semantic-segmentation-on-1","task":"Open Vocabulary Semantic Segmentation","dataset":"PASCAL Context-59","model":"TaAlign(trained with image-text pairs)","rank_in_archive_order":19,"of":24,"metrics":{"mIoU":"37.6"},"uses_additional_data":false},{"leaderboard":"/sota/open-vocabulary-semantic-segmentation-on-5","task":"Open Vocabulary Semantic Segmentation","dataset":"PascalVOC-20","model":"TagAlign(trained with image-text pairs)","rank_in_archive_order":14,"of":20,"metrics":{"mIoU":"87.9"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-semantic-segmentation-with-4","task":"Unsupervised Semantic Segmentation with Language-image Pre-training","dataset":"ADE20K","model":"TagAlign","rank_in_archive_order":6,"of":13,"metrics":{"Mean IoU (val)":"17.3"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-semantic-segmentation-with-10","task":"Unsupervised Semantic Segmentation with Language-image Pre-training","dataset":"COCO-Object","model":"TagAlign","rank_in_archive_order":6,"of":12,"metrics":{"mIoU":"33.3"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-semantic-segmentation-with-9","task":"Unsupervised Semantic Segmentation with Language-image Pre-training","dataset":"COCO-Stuff-171","model":"TagAlign","rank_in_archive_order":5,"of":12,"metrics":{"mIoU":"25.3"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-semantic-segmentation-with-3","task":"Unsupervised Semantic Segmentation with Language-image Pre-training","dataset":"Cityscapes val","model":"TagAlign","rank_in_archive_order":6,"of":12,"metrics":{"mIoU":"27.5"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-semantic-segmentation-with-8","task":"Unsupervised Semantic Segmentation with Language-image Pre-training","dataset":"PASCAL Context-59","model":"TagAlign","rank_in_archive_order":5,"of":12,"metrics":{"mIoU":"37.6"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-semantic-segmentation-with-11","task":"Unsupervised Semantic Segmentation with Language-image Pre-training","dataset":"PASCAL VOC","model":"TagAlign","rank_in_archive_order":8,"of":10,"metrics":{"mIoU":"53.9"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-semantic-segmentation-with-7","task":"Unsupervised Semantic Segmentation with Language-image Pre-training","dataset":"PascalVOC-20","model":"TagAlign","rank_in_archive_order":4,"of":10,"metrics":{"mIoU":"87.9"},"uses_additional_data":false}],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}