Papers › Self-supervised Knowledge Distillation for Few-shot Learning

Self-supervised Knowledge Distillation for Few-shot Learning

17 Jun 2020arXiv:2006.09785archive 2025-07-28

Jathushan Rajasegaran, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Mubarak Shah

Real-world contains an overwhelmingly large number of object classes, learning all of which at once is infeasible. Few shot learning is a promising learning paradigm due to its ability to learn out of order distributions quickly with only a few samples. Recent works [7, 41] show that simply learning a good feature embedding can outperform more sophisticated meta-learning and metric learning algorithms for few-shot learning. In this paper, we propose a simple approach to improve the representation capacity of deep neural networks for few-shot learning tasks. We follow a two-stage learning process: First, we train a neural network to maximize the entropy of the feature embedding, thus creating an optimal output manifold using a self-supervised auxiliary loss. In the second stage, we minimize the entropy on feature embedding by bringing self-supervised twins together, while constraining the manifold with student-teacher distillation. Our experiments show that, even in the first stage, self-supervision can outperform current state-of-the-art methods, with further gains achieved by our second stage distillation process. Our codes are available at: https://github.com/brjathu/SKD.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

brjathu/SKD officialmentioned in papermentioned on GitHubpytorchSyntology: not harvested report
yiren-jian/embedding-learning-fsl mentioned on GitHubpytorchSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Few-Shot Image ClassificationFew-Shot LearningKnowledge DistillationMeta-LearningMetric Learning

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Few-Shot Image Classification CIFAR-FS 5-way (1-shot) SKD Accuracy 76.9 #19 of 38 Archive leaderboard report
Few-Shot Image Classification CIFAR-FS 5-way (5-shot) SKD Accuracy 88.9 #17 of 39 Archive leaderboard report
Few-Shot Image Classification FC100 5-way (1-shot) SKD Accuracy 46.5 #13 of 22 Archive leaderboard report
Few-Shot Image Classification FC100 5-way (5-shot) SKD Accuracy 63.1 #13 of 22 Archive leaderboard report
Few-Shot Image Classification Mini-Imagenet 5-way (1-shot) SKD Accuracy 67.04 #50 of 105 Archive leaderboard report
Few-Shot Image Classification Mini-Imagenet 5-way (5-shot) SKD Accuracy 83.54 #34 of 95 Archive leaderboard report
Few-Shot Image Classification Tiered ImageNet 5-way (1-shot) SKD Accuracy 72.03 #27 of 49 Archive leaderboard report
Few-Shot Image Classification Tiered ImageNet 5-way (5-shot) SKD Accuracy 86.66 #24 of 51 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections