Papers › Simple Semi-supervised Knowledge Distillation from Vision-Language Models via...
Simple Semi-supervised Knowledge Distillation from Vision-Language Models via 𝐃ual-𝐇ead 𝐎ptimization
Seongjae Kang, Dong Bok Lee, Hyungjoon Jang, Sung Ju Hwang
Vision-language models (VLMs) have achieved remarkable success across diverse tasks by leveraging rich textual information with minimal labeled data. However, deploying such large models remains challenging, particularly in resource-constrained environments. Knowledge distillation (KD) offers a well-established solution to this problem; however, recent KD approaches from VLMs often involve multi-stage training or additional tuning, increasing computational overhead and optimization complexity. In this paper, we propose 𝐃ual-𝐇ead 𝐎ptimization (𝐃𝐇𝐎) -- a simple yet effective KD framework that transfers knowledge from VLMs to compact, task-specific models in semi-supervised settings. Specifically, we introduce dual prediction heads that independently learn from labeled data and teacher predictions, and propose to linearly combine their outputs during inference. We observe that DHO mitigates gradient conflicts between supervised and distillation signals, enabling more effective feature learning than single-head KD baselines. As a result, extensive experiments show that DHO consistently outperforms baselines across multiple domains and fine-grained datasets. Notably, on ImageNet, it achieves state-of-the-art performance, improving accuracy by 3% and 0.1% with 1% and 10% labeled data, respectively, while using fewer parameters.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
1 archive task tag without a task page not shown.
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Semi-Supervised Image Classification | ImageNet - 1% labeled data | DHO (ViT-Large) | Top 1 Accuracy | 84.6% | #1 of 65 | Archive leaderboard | report |
| Semi-Supervised Image Classification | ImageNet - 1% labeled data | DHO (ViT-Base) | Top 1 Accuracy | 81.6% | #3 of 65 | Archive leaderboard | report |
| Semi-Supervised Image Classification | ImageNet - 10% labeled data | DHO (ViT-Large) | Top 1 Accuracy | 85.9% | #1 of 75 | Archive leaderboard | report |
| Semi-Supervised Image Classification | ImageNet - 10% labeled data | DHO (ViT-Base) | Top 1 Accuracy | 82.8% | #8 of 75 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections