{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dinov2-meets-text-a-unified-framework-for","title":"DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment","arxiv_id":"2412.16334","date":"2024-12-20","proceeding":"CVPR 2025 1","authors":["Cijo Jose","Théo Moutakanni","Dahyun Kang","Federico Baldassarre","Timothée Darcet","Hu Xu","Daniel Li","Marc Szafraniec","Michaël Ramamonjisoa","Maxime Oquab","Oriane Siméoni","Huy V. Vo","Patrick Labatut","Piotr Bojanowski"],"abstract":"Self-supervised visual foundation models produce powerful embeddings that achieve remarkable performance on a wide range of downstream tasks. However, unlike vision-language models such as CLIP, self-supervised visual features are not readily aligned with language, hindering their adoption in open-vocabulary tasks. Our method, named dino.txt, unlocks this new ability for DINOv2, a widely used self-supervised visual encoder. We build upon the LiT training strategy, which trains a text encoder to align with a frozen vision model but leads to unsatisfactory results on dense tasks. We propose several key ingredients to improve performance on both global and dense tasks, such as concatenating the [CLS] token with the patch average to train the alignment and curating data using both text and image modalities. With these, we successfully train a CLIP-like model with only a fraction of the computational cost compared to CLIP while achieving state-of-the-art results in zero-shot classification and open-vocabulary semantic segmentation.","url_abs":"https://arxiv.org/abs/2412.16334v1","url_pdf":"https://arxiv.org/pdf/2412.16334v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dinov2-meets-text-a-unified-framework-for","repo_url":"https://github.com/facebookresearch/dinov2","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"open-vocabulary-semantic-segmentation","task_name":"Open Vocabulary Semantic Segmentation"},{"task_slug":"open-vocabulary-semantic-segmentation-1","task_name":"Open-Vocabulary Semantic Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"},{"task_slug":null,"task_name":"zero-shot-classification"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.16334","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}