Papers › A Fast Knowledge Distillation Framework for Visual Recognition

A Fast Knowledge Distillation Framework for Visual Recognition

2 Dec 2021arXiv:2112.01528archive 2025-07-28

Zhiqiang Shen, Eric Xing

While Knowledge Distillation (KD) has been recognized as a useful tool in many visual tasks, such as supervised classification and self-supervised representation learning, the main drawback of a vanilla KD framework is its mechanism, which consumes the majority of the computational overhead on forwarding through the giant teacher networks, making the entire learning procedure inefficient and costly. ReLabel, a recently proposed solution, suggests creating a label map for the entire image. During training, it receives the cropped region-level label by RoI aligning on a pre-generated entire label map, allowing for efficient supervision generation without having to pass through the teachers many times. However, as the KD teachers are from conventional multi-crop training, there are various mismatches between the global label-map and region-level label in this technique, resulting in performance deterioration. In this study, we present a Fast Knowledge Distillation (FKD) framework that replicates the distillation training phase and generates soft labels using the multi-crop KD approach, while training faster than ReLabel since no post-processes such as RoI align and softmax operations are used. When conducting multi-crop in the same image for data loading, our FKD is even more efficient than the traditional image classification framework. On ImageNet-1K, we obtain 79.8% with ResNet-50, outperforming ReLabel by ~1.0% while being faster. On the self-supervised learning task, we also show that FKD has an efficiency advantage. Our project page: http://zhiqiangshen.com/projects/FKD/index.html, source code and models are available at: https://github.com/szq0214/FKD.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

szq0214/fkd officialmentioned in papermentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to EfficientNetV2: Smaller Models and Faster Training (arXiv:2104.00298; that paper's own run record: 55 ran (of which 14 constructed an object rather than computing a result; 43 with no instrument failure: 11 honoured, 0 violated, 32 with no contract checked; 12 where Syntology's instrument failed) · 24 unverified) and FerKD: Surgical Label Adaptation for Efficient Distillation (arXiv:2312.17473; that paper's own run record: 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report
szq0214/MEAL-V2 mentioned on GitHubpytorchSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Image ClassificationKnowledge DistillationRepresentation LearningSelf-Supervised Learningimage-classification

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Image Classification ImageNet ResNet-101 (224 res, Fast Knowledge Distillation) Top 1 Accuracy 81.9% #594 of 1060 Archive leaderboard report
Image Classification ImageNet ResNet-50 (224 res, Fast Knowledge Distillation) Top 1 Accuracy 80.1% #716 of 1060 Archive leaderboard report
Image Classification ImageNet SReT-LT (Fast Knowledge Distillation) GFLOPs 1.2 #812 of 1060 Archive leaderboard report
Image Classification ImageNet SReT-LT (Fast Knowledge Distillation) Number of params 5M #812 of 1060 Archive leaderboard report
Image Classification ImageNet SReT-LT (Fast Knowledge Distillation) Top 1 Accuracy 78.7% #812 of 1060 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Knowledge DistillationSoftmax

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections