Papers › Self-supervised Knowledge Distillation for Few-shot Learning

Self-supervised Knowledge Distillation for Few-shot Learning

17 Jun 2020arXiv:2006.09785archive 2025-07-28

Jathushan Rajasegaran, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Mubarak Shah

Real-world contains an overwhelmingly large number of object classes, learning all of which at once is infeasible. Few shot learning is a promising learning paradigm due to its ability to learn out of order distributions quickly with only a few samples. Recent works [7, 41] show that simply learning a good feature embedding can outperform more sophisticated meta-learning and metric learning algorithms for few-shot learning. In this paper, we propose a simple approach to improve the representation capacity of deep neural networks for few-shot learning tasks. We follow a two-stage learning process: First, we train a neural network to maximize the entropy of the feature embedding, thus creating an optimal output manifold using a self-supervised auxiliary loss. In the second stage, we minimize the entropy on feature embedding by bringing self-supervised twins together, while constraining the manifold with student-teacher distillation. Our experiments show that, even in the first stage, self-supervision can outperform current state-of-the-art methods, with further gains achieved by our second stage distillation process. Our codes are available at: https://github.com/brjathu/SKD.

Paper PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

brjathu/SKD officialmentioned in papermentioned on GitHubpytorch report

yiren-jian/embedding-learning-fsl mentioned on GitHubpytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Few-Shot Image Classification Few-Shot Learning Knowledge Distillation Meta-Learning Metric Learning

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Few-Shot Image Classification	CIFAR-FS 5-way (1-shot)	SKD	Accuracy	76.9	#19 of 38	Archive leaderboard	report
Few-Shot Image Classification	CIFAR-FS 5-way (5-shot)	SKD	Accuracy	88.9	#17 of 39	Archive leaderboard	report
Few-Shot Image Classification	FC100 5-way (1-shot)	SKD	Accuracy	46.5	#13 of 22	Archive leaderboard	report
Few-Shot Image Classification	FC100 5-way (5-shot)	SKD	Accuracy	63.1	#13 of 22	Archive leaderboard	report
Few-Shot Image Classification	Mini-Imagenet 5-way (1-shot)	SKD	Accuracy	67.04	#50 of 105	Archive leaderboard	report
Few-Shot Image Classification	Mini-Imagenet 5-way (5-shot)	SKD	Accuracy	83.54	#34 of 95	Archive leaderboard	report
Few-Shot Image Classification	Tiered ImageNet 5-way (1-shot)	SKD	Accuracy	72.03	#27 of 49	Archive leaderboard	report
Few-Shot Image Classification	Tiered ImageNet 5-way (5-shot)	SKD	Accuracy	86.66	#24 of 51	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections