{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/your-diffusion-model-is-secretly-a-zero-shot","title":"Your Diffusion Model is Secretly a Zero-Shot Classifier","arxiv_id":"2303.16203","date":"2023-03-28","proceeding":"ICCV 2023 1","authors":["Alexander C. Li","Mihir Prabhudesai","Shivam Duggal","Ellis Brown","Deepak Pathak"],"abstract":"The recent wave of large-scale text-to-image diffusion models has dramatically increased our text-based image generation abilities. These models can generate realistic images for a staggering variety of prompts and exhibit impressive compositional generalization abilities. Almost all use cases thus far have solely focused on sampling; however, diffusion models can also provide conditional density estimates, which are useful for tasks beyond image generation. In this paper, we show that the density estimates from large-scale text-to-image diffusion models like Stable Diffusion can be leveraged to perform zero-shot classification without any additional training. Our generative approach to classification, which we call Diffusion Classifier, attains strong results on a variety of benchmarks and outperforms alternative methods of extracting knowledge from diffusion models. Although a gap remains between generative and discriminative approaches on zero-shot recognition tasks, our diffusion-based approach has significantly stronger multimodal compositional reasoning ability than competing discriminative approaches. Finally, we use Diffusion Classifier to extract standard classifiers from class-conditional diffusion models trained on ImageNet. Our models achieve strong classification performance using only weak augmentations and exhibit qualitatively better \"effective robustness\" to distribution shift. Overall, our results are a step toward using generative over discriminative models for downstream tasks. Results and visualizations at https://diffusion-classifier.github.io/","url_abs":"https://arxiv.org/abs/2303.16203v3","url_pdf":"https://arxiv.org/pdf/2303.16203v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"your-diffusion-model-is-secretly-a-zero-shot","repo_url":"https://github.com/diffusion-classifier/diffusion-classifier","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"your-diffusion-model-is-secretly-a-zero-shot","repo_url":"https://github.com/LiYinqi/DIVE","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"your-diffusion-model-is-secretly-a-zero-shot","repo_url":"https://github.com/SamsungSAILMontreal/ForestDiffusion","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"your-diffusion-model-is-secretly-a-zero-shot","repo_url":"https://github.com/tajamul21/fate","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}}],"tasks":[{"task_slug":"domain-generalization","task_name":"Domain Generalization"},{"task_slug":"fine-grained-image-classification","task_name":"Fine-Grained Image Classification"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"relational-reasoning","task_name":"Relational Reasoning"},{"task_slug":"visual-reasoning","task_name":"Visual Reasoning"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"},{"task_slug":"zero-shot-transfer-image-classification","task_name":"Zero-Shot Transfer Image Classification"},{"task_slug":null,"task_name":"zero-shot-classification"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/domain-generalization-on-imagenet-a","task":"Domain Generalization","dataset":"ImageNet-A","model":"Diffusion Classifier","rank_in_archive_order":27,"of":39,"metrics":{"Top-1 accuracy %":"30.2"},"uses_additional_data":false},{"leaderboard":"/sota/fine-grained-image-classification-on-fgvc","task":"Fine-Grained Image Classification","dataset":"FGVC Aircraft","model":"Diffusion Classifier (zero-shot)","rank_in_archive_order":55,"of":57,"metrics":{"Accuracy":"26.4"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-cifar-10","task":"Image Classification","dataset":"CIFAR-10","model":"Diffusion Classifier (zero-shot)","rank_in_archive_order":217,"of":265,"metrics":{"Percentage correct":"88.5"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-flowers-102","task":"Image Classification","dataset":"Flowers-102","model":"Diffusion Classifier (zero-shot)","rank_in_archive_order":51,"of":52,"metrics":{"Per-Class Accuracy":"66.3"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"Diffusion Classifier","rank_in_archive_order":779,"of":1060,"metrics":{"Top 1 Accuracy":"79.1%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-objectnet-imagenet","task":"Image Classification","dataset":"ObjectNet (ImageNet classes)","model":"Diffusion Classifier (zero-shot)","rank_in_archive_order":1,"of":2,"metrics":{"Top 1 Accuracy":"43.4"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-objectnet-imagenet","task":"Image Classification","dataset":"ObjectNet (ImageNet classes)","model":"Diffusion Classifier","rank_in_archive_order":2,"of":2,"metrics":{"Top 1 Accuracy":"33.9"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-oxford-iiit-pets-1","task":"Image Classification","dataset":"Oxford-IIIT Pets","model":"Diffusion Classifier (zero-shot)","rank_in_archive_order":6,"of":6,"metrics":{"Per-Class Accuracy":"87.3"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-stl-10","task":"Image Classification","dataset":"STL-10","model":"Diffusion Classifier (zero-shot)","rank_in_archive_order":18,"of":117,"metrics":{"Percentage correct":"95.4"},"uses_additional_data":false},{"leaderboard":"/sota/visual-reasoning-on-winoground","task":"Visual Reasoning","dataset":"Winoground","model":"Diffusion Classifier (zero-shot)","rank_in_archive_order":56,"of":114,"metrics":{"Text Score":"34.00"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-transfer-image-classification-on-17","task":"Zero-Shot Transfer Image Classification","dataset":"Food-101","model":"Diffusion Classifier (zero-shot)","rank_in_archive_order":5,"of":5,"metrics":{"Top 1 Accuracy":"77.7"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-transfer-image-classification-on-1","task":"Zero-Shot Transfer Image Classification","dataset":"ImageNet","model":"Diffusion Classifier (zero-shot)","rank_in_archive_order":20,"of":23,"metrics":{"Accuracy (Private)":"61.4"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2303.16203","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.16203"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/SamsungSAILMontreal/ForestDiffusion","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/diffusion-classifier/diffusion-classifier","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tajamul21/fate","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/LiYinqi/DIVE","reach":null}],"summary":{"ran_honours":1,"ran_fixture":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"5236fd023ace5c38","entry":"mean_per_class_acc","repo":"diffusion-classifier/diffusion-classifier","repo_kind":"official","path":"scripts/print_acc.py","file_url":"https://github.com/diffusion-classifier/diffusion-classifier/blob/HEAD/scripts/print_acc.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5236fd023ace5c38"}},{"code_sha256_prefix":"9fbea808db47dc66","entry":"project_logits","repo":"LiYinqi/DIVE","repo_kind":"listed","path":"Classification/diffusion/dataset/imagenet.py","file_url":"https://github.com/LiYinqi/DIVE/blob/HEAD/Classification/diffusion/dataset/imagenet.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9fbea808db47dc66"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}