Papers › SpectralKD: A Unified Framework for Interpreting and Distilling Vision Transformers...
SpectralKD: A Unified Framework for Interpreting and Distilling Vision Transformers via Spectral Analysis
Huiyuan Tian, Bonan Xu, Shijian Li, Gang Pan
Knowledge Distillation (KD) has achieved widespread success in compressing large Vision Transformers (ViTs), but a unified theoretical framework for both ViTs and KD is still lacking. In this paper, we propose SpectralKD, a novel unified analytical framework that offers deeper insights into ViTs and optimizes KD via spectral analysis. Our model-wise analysis reveals that CaiT concentrates information in their first and last few layers, informing optimal layer selection for KD. Surprisingly, our layer-wise analysis discovers that Swin Transformer and CaiT exhibit similar spectral encoding patterns despite their architectural differences, leading to feature map alignment guideline. Building on these insights, we propose a simple yet effective spectral alignment method for KD. Benefiting from the deeper understanding by above analysis results, even such a simple strategy achieves state-of-the-art performance on ImageNet-1K without introducing any trainable parameters, improving DeiT-Tiny by +5.2% and Swin-Tiny by +1.4% in top-1 accuracy. Furthermore, our post-training analysis reveals that distilled students can reproduce spectral patterns similar to their teachers, opening a new area we term ``distillation dynamics". Code and experimental logs are available in https://github.com/thy960112/SpectralKD.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Knowledge Distillation | ImageNet | SpectralKD (T:Swin-S S:Swin-T) | CRD training setting | ✘ | #7 of 52 | Archive leaderboard | report |
| Knowledge Distillation | ImageNet | SpectralKD (T:Swin-S S:Swin-T) | Top-1 accuracy % | 82.7 | #7 of 52 | Archive leaderboard | report |
| Knowledge Distillation | ImageNet | SpectralKD (T:Swin-S S:Swin-T) | model size | 22M | #7 of 52 | Archive leaderboard | report |
| Knowledge Distillation | ImageNet | SpectralKD (T:Cait-S24 S:DeiT-S) | CRD training setting | ✘ | #11 of 52 | Archive leaderboard | report |
| Knowledge Distillation | ImageNet | SpectralKD (T:Cait-S24 S:DeiT-S) | Top-1 accuracy % | 82.2 | #11 of 52 | Archive leaderboard | report |
| Knowledge Distillation | ImageNet | SpectralKD (T:Cait-S24 S:DeiT-S) | model size | 22M | #11 of 52 | Archive leaderboard | report |
| Knowledge Distillation | ImageNet | SpectralKD (T:Cait-S24 S:DeiT-T) | CRD training setting | ✘ | #20 of 52 | Archive leaderboard | report |
| Knowledge Distillation | ImageNet | SpectralKD (T:Cait-S24 S:DeiT-T) | Top-1 accuracy % | 77.4 | #20 of 52 | Archive leaderboard | report |
| Knowledge Distillation | ImageNet | SpectralKD (T:Cait-S24 S:DeiT-T) | model size | 6M | #20 of 52 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections