Papers › PromptKD: Unsupervised Prompt Distillation for Vision-Language Models

PromptKD: Unsupervised Prompt Distillation for Vision-Language Models

5 Mar 2024CVPR 2024 1arXiv:2403.02781archive 2025-07-28

Zheng Li, Xiang Li, Xinyi Fu, Xin Zhang, Weiqiang Wang, Shuo Chen, Jian Yang

Prompt learning has emerged as a valuable technique in enhancing vision-language models (VLMs) such as CLIP for downstream tasks in specific domains. Existing work mainly focuses on designing various learning forms of prompts, neglecting the potential of prompts as effective distillers for learning from larger teacher models. In this paper, we introduce an unsupervised domain prompt distillation framework, which aims to transfer the knowledge of a larger teacher model to a lightweight target model through prompt-driven imitation using unlabeled domain images. Specifically, our framework consists of two distinct stages. In the initial stage, we pre-train a large CLIP teacher model using domain (few-shot) labels. After pre-training, we leverage the unique decoupled-modality characteristics of CLIP by pre-computing and storing the text features as class vectors only once through the teacher text encoder. In the subsequent stage, the stored class vectors are shared across teacher and student image encoders for calculating the predicted logits. Further, we align the logits of both the teacher and student models via KL divergence, encouraging the student image encoder to generate similar probability distributions to the teacher through the learnable prompts. The proposed prompt distillation process eliminates the reliance on labeled data, enabling the algorithm to leverage a vast amount of unlabeled images within the domain. Finally, the well-trained student image encoders and pre-stored text features (class vectors) are utilized for inference. To our best knowledge, we are the first to (1) perform unsupervised domain-specific prompt-driven knowledge distillation for CLIP, and (2) establish a practical pre-storing mechanism of text features as shared class vectors between teacher and student. Extensive experiments on 11 datasets demonstrate the effectiveness of our method.

PaperPDFConference PDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

zhengli97/promptkd officialmentioned in papermentioned on GitHubpytorchApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to ATPrompt: Textual Prompt Learning with Embedded Attributes (arXiv:2412.09442; that paper's own run record: 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Knowledge DistillationPrompt EngineeringPrompt LearningZero-Shot Image Classification

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Prompt Engineering Caltech-101 PromptKD Harmonic mean 97.77 #1 of 14 Archive leaderboard report
Prompt Engineering DTD PromptKD Harmonic mean 77.94 #1 of 14 Archive leaderboard report
Prompt Engineering EuroSAT PromptKD Harmonic mean 89.14 #2 of 14 Archive leaderboard report
Prompt Engineering FGVC-Aircraft PromptKD Harmonic mean 45.17 #1 of 14 Archive leaderboard report
Prompt Engineering Food-101 PromptKD Harmonic mean 93.05 #1 of 13 Archive leaderboard report
Prompt Engineering ImageNet PromptKD Harmonic mean 77.62 #1 of 15 Archive leaderboard report
Prompt Engineering Oxford 102 Flower PromptKD Harmonic mean 90.24 #1 of 14 Archive leaderboard report
Prompt Engineering Oxford-IIIT Pet Dataset PromptKD Harmonic mean 97.15 #1 of 14 Archive leaderboard report
Prompt Engineering SUN397 PromptKD Harmonic mean 82.60 #1 of 14 Archive leaderboard report
Prompt Engineering Stanford Cars PromptKD Harmonic mean 83.13 #1 of 14 Archive leaderboard report
Prompt Engineering UCF101 PromptKD Harmonic mean 86.10 #1 of 14 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

CLIPKnowledge Distillation

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections