Knowledge Distillation

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

11 leaderboard tables shown for this task, 7 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted. 10 shown of 11 until expanded.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

6 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

2 subtasks in the archive's task tree.

Most implemented papers archive 2025-07-28

30 shown of 1,740 papers with code (4,240 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• Focal Loss for Dense Object Detection
  7 Aug 2017 234 repositories listed Syntology ran 11 of 11 samples · 0 unverified · 6 pointer-only (licence)
  Our novel Focal Loss focuses training on a sparse set of hard examples and prevents the vast number of easy negatives from overwhelming the detector during training.
• Distilling the Knowledge in a Neural Network
  9 Mar 2015 64 repositories listed Syntology ran 15 of 37 samples · 22 unverified · 10 pointer-only (licence)
  A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.
• Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
  23 Aug 2019 40 repositories listed Syntology ran 6 of 32 samples · 26 unverified
  Recent developments in natural language representations have been accompanied by large and expensive models that leverage vast amounts of general-domain text through self-supervised pre-training.
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
  8 Jun 2020 37 repositories listed Syntology ran 73 of 119 samples · 46 unverified · 33 pointer-only (licence)
  In this paper, we propose FastSpeech 2, which addresses the issues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the…
• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
  2 Oct 2019 37 repositories listed Syntology ran 19 of 27 samples · 8 unverified
  As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or…
• Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks
  30 Oct 2017 24 repositories listed Syntology ran 4 of 7 samples · 3 unverified · 3 pointer-only (licence)
  Over the last decade, Convolutional Neural Network (CNN) models have been highly successful in solving complex vision problems.
• Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
  21 Apr 2020 12 repositories listed Syntology ran 0 of 1 samples · 1 unverified
  The training is based on the idea that a translated sentence should be mapped to the same location in the vector space as the original sentence.
• Knowledge distillation: A good teacher is patient and consistent
  9 Jun 2021 10 repositories listed
  In particular, we uncover that there are certain implicit design choices, which may drastically affect the effectiveness of distillation.
• TinyBERT: Distilling BERT for Natural Language Understanding
  23 Sep 2019 10 repositories listed Syntology ran 0 of 4 samples · 4 unverified · 4 pointer-only (licence)
  To accelerate inference and reduce model size while maintaining accuracy, we first propose a novel Transformer distillation method that is specially designed for knowledge distillation (KD) of the Transformer-based…
• AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time
  7 Nov 2022 8 repositories listed
  Accurate whole-body multi-person pose estimation and tracking is an important yet challenging topic in computer vision.
• Distilling Knowledge via Knowledge Review
  19 Apr 2021 7 repositories listed Syntology ran 5 of 8 samples · 3 unverified · 5 pointer-only (licence)
  Knowledge distillation transfers knowledge from the teacher network to the student one, with the goal of greatly improving the performance of the student network.
• FedMD: Heterogenous Federated Learning via Model Distillation
  8 Oct 2019 7 repositories listed Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
  With 10 distinct participants, the final test accuracy of each model on average receives a 20% gain on top of what's possible without collaboration and is only a few percent lower than the performance each model would…
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
  12 Dec 2016 7 repositories listed Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
  Attention plays a critical role in human visual experience.
• Sequence-Level Knowledge Distillation
  25 Jun 2016 6 repositories listed Syntology ran 0 of 2 samples · 2 unverified
  We demonstrate that standard knowledge distillation applied to word-level prediction can be effective for NMT, and also introduce two novel sequence-level versions of knowledge distillation that further improve…
• Awesome Multi-modal Object Tracking
  23 May 2024 5 repositories listed
  To leverage more modalities, some recent efforts have been made to learn a unified visual object tracking model for any modality.
• DistiLLM: Towards Streamlined Distillation for Large Language Models
  6 Feb 2024 5 repositories listed Syntology ran 5 of 5 samples · 0 unverified · 4 pointer-only (licence)
  Knowledge distillation (KD) is widely used for compressing a teacher model to a smaller student model, reducing its inference cost and memory footprint while preserving model capabilities.
• Anomaly Detection via Reverse Distillation from One-Class Embedding
  26 Jan 2022 5 repositories listed Syntology ran 14 of 18 samples · 4 unverified · 15 pointer-only (licence)
  Knowledge distillation (KD) achieves promising results on the challenging problem of unsupervised anomaly detection (AD).
• TernaryBERT: Distillation-aware Ultra-low Bit BERT
  27 Sep 2020 5 repositories listed
  Transformer-based pre-training models like BERT have achieved remarkable performance in many natural language processing tasks.
• ProSelfLC: Progressive Self Label Correction for Training Robust Deep Neural Networks
  7 May 2020 5 repositories listed Syntology ran 1 of 11 samples · 10 unverified
  Keywords: entropy minimisation, maximum entropy, confidence penalty, self knowledge distillation, label correction, label noise, semi-supervised learning, output regularisation
• Patient Knowledge Distillation for BERT Model Compression
  25 Aug 2019 5 repositories listed Syntology ran 9 of 27 samples · 18 unverified · 27 pointer-only (licence)
  Pre-trained language models such as BERT have proven to be highly effective for natural language processing (NLP) tasks.
• BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
  5 Feb 2024 4 repositories listed Syntology ran 3 of 14 samples · 11 unverified
  It can simultaneously perform the three common retrieval functionalities of embedding model: dense retrieval, multi-vector retrieval, and sparse retrieval, which provides a unified model foundation for real-world IR…
• Event Stream-based Visual Object Tracking: A High-Resolution Benchmark Dataset and A Novel Baseline
  26 Sep 2023 4 repositories listed Syntology ran 9 of 9 samples · 0 unverified · 9 pointer-only (licence)
  Tracking using bio-inspired event cameras has drawn more and more attention in recent years.
• Orca: Progressive Learning from Complex Explanation Traces of GPT-4
  5 Jun 2023 4 repositories listed
  To address these challenges, we develop Orca (We are working with our legal team to publicly release a diff of the model weights in accordance with LLaMA's release policy to be published at https://aka.
• Decoupled Kullback-Leibler Divergence Loss
  23 May 2023 4 repositories listed Syntology ran 7 of 11 samples · 4 unverified · 4 pointer-only (licence)
  In this paper, we delve deeper into the Kullback-Leibler (KL) Divergence loss and mathematically prove that it is equivalent to the Decoupled Kullback-Leibler (DKL) Divergence loss that consists of 1) a weighted Mean…
• Understanding the Role of the Projector in Knowledge Distillation
  20 Mar 2023 4 repositories listed
  We then show that the normalisation of representations is tightly coupled with the training dynamics of this projector, which can have a large impact on the students performance.
• Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?
  16 Dec 2022 4 repositories listed
  The success of deep learning heavily relies on large-scale data with comprehensive labels, which is more expensive and time-consuming to fetch in 3D compared to 2D images or natural languages.
• SA-MLP: Distilling Graph Knowledge from GNNs into Structure-Aware MLP
  18 Oct 2022 4 repositories listed Syntology ran 0 of 5 samples · 5 unverified
  Then, we introduce a novel structure-mixing knowledge distillation strategy to enhance the learning ability of MLPs for structure information.
• ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech
  13 Jul 2022 4 repositories listed Syntology ran 2 of 6 samples · 4 unverified
  Through the preliminary study on diffusion model parameterization, we find that previous gradient-based TTS models require hundreds or thousands of iterations to guarantee high sample quality, which poses a challenge…
• Structured Sparse R-CNN for Direct Scene Graph Generation
  21 Jun 2021 4 repositories listed Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
  The key to our method is a set of learnable triplet queries and a structured triplet detector which could be jointly optimized from the training set in an end-to-end manner.
• Data-Free Knowledge Distillation for Heterogeneous Federated Learning
  20 May 2021 4 repositories listed
  Federated Learning (FL) is a decentralized machine-learning paradigm, in which a global server iteratively averages the model parameters of local users without accessing their data.

Syntology lines on 22 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.