{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/diffusion-feedback-helps-clip-see-better","title":"Diffusion Feedback Helps CLIP See Better","arxiv_id":"2407.20171","date":"2024-07-29","proceeding":null,"authors":["Wenxuan Wang","Quan Sun","Fan Zhang","Yepeng Tang","Jing Liu","Xinlong Wang"],"abstract":"Contrastive Language-Image Pre-training (CLIP), which excels at abstracting open-world representations across domains and modalities, has become a foundation for a variety of vision and multimodal tasks. However, recent studies reveal that CLIP has severe visual shortcomings, such as which can hardly distinguish orientation, quantity, color, structure, etc. These visual shortcomings also limit the perception capabilities of multimodal large language models (MLLMs) built on CLIP. The main reason could be that the image-text pairs used to train CLIP are inherently biased, due to the lack of the distinctiveness of the text and the diversity of images. In this work, we present a simple post-training approach for CLIP models, which largely overcomes its visual shortcomings via a self-supervised diffusion process. We introduce DIVA, which uses the DIffusion model as a Visual Assistant for CLIP. Specifically, DIVA leverages generative feedback from text-to-image diffusion models to optimize CLIP representations, with only images (without corresponding text). We demonstrate that DIVA improves CLIP's performance on the challenging MMVP-VLM benchmark which assesses fine-grained visual abilities to a large extent (e.g., 3-7%), and enhances the performance of MLLMs and vision models on multimodal understanding and segmentation tasks. Extensive evaluation on 29 image classification and retrieval benchmarks confirms that our framework preserves CLIP's strong zero-shot capabilities. The code is available at https://github.com/baaivision/DIVA.","url_abs":"https://arxiv.org/abs/2407.20171v4","url_pdf":"https://arxiv.org/pdf/2407.20171v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"diffusion-feedback-helps-clip-see-better","repo_url":"https://github.com/baaivision/diva","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.20171","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.20171"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/baaivision/diva","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":2,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f23f374916d14243","entry":"checkpoint_filter_fn","repo":"baaivision/diva","repo_kind":"official","path":"condition/SigLIP_for_timm_models_visiontransformer.py","file_url":"https://github.com/baaivision/diva/blob/HEAD/condition/SigLIP_for_timm_models_visiontransformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f23f374916d14243"}},{"code_sha256_prefix":"42d148d34d7586d4","entry":"get_init_weights_vit","repo":"baaivision/diva","repo_kind":"official","path":"condition/SigLIP_for_timm_models_visiontransformer.py","file_url":"https://github.com/baaivision/diva/blob/HEAD/condition/SigLIP_for_timm_models_visiontransformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"42d148d34d7586d4"}},{"code_sha256_prefix":"ab308f362b0e9ee9","entry":"build_model","repo":"baaivision/diva","repo_kind":"official","path":"condition/OpenAICLIP_for_clip_model.py","file_url":"https://github.com/baaivision/diva/blob/HEAD/condition/OpenAICLIP_for_clip_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ab308f362b0e9ee9"}},{"code_sha256_prefix":"ed04551ca8295bde","entry":"resize_pos_embed","repo":"baaivision/diva","repo_kind":"official","path":"condition/SigLIP_for_timm_models_visiontransformer.py","file_url":"https://github.com/baaivision/diva/blob/HEAD/condition/SigLIP_for_timm_models_visiontransformer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed04551ca8295bde"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}