Browse State-of-the-Art › Data-free Knowledge Distillation
Data-free Knowledge Distillation
37 papers with code · 2 benchmarks · 3 datasets archive 2025-07-28
Benchmarks archive 2025-07-28
2 leaderboard tables shown for this task, 2 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.
| Dataset | Best model (first row in archive order) | Paper | Code | Syntology | Compare |
|---|---|---|---|---|---|
| QNLI (4 rows) | GOLD (T5-base) | GOLD: Generalized Knowledge Distillation via... | — | — | Compare |
| SQuAD (4 rows) | GOLD (T5-base) | GOLD: Generalized Knowledge Distillation via... | — | — | Compare |
Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
3 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
1 subtask in the archive's task tree.
Parent tasks archive 2025-07-28
Papers archive 2025-07-28
archive papers tagged: 75 · with a code link: 37 · where Syntology ran a sample: 10 (6 with a run with no instrument failure, 4 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (10 of 75 tagged: 6 with a run with no instrument failure, 4 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 75 of 75, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
20 May 2021 4 repositories listedFederated Learning (FL) is a decentralized machine-learning paradigm, in which a global server iteratively averages the model parameters of local users without accessing their data.
-
16 Feb 2022 3 repositories listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 4 unverified (of 7 harvested samples) · 7 pointer-only (licence)There is a growing interest in dataset generation recently due to the superior generative capacity of large pre-trained language models (PLMs).
-
10 Sep 2023 2 repositories listedWith the increasing deployment of deep neural networks in safety-critical applications such as self-driving cars, medical imaging, anomaly detection, etc., adversarial robustness has become a crucial concern in the…
-
22 Oct 2022 2 repositories listedTo improve the quality of dataset synthesis, we propose a progressive zero-shot dataset generation framework, ProGen, which leverages the feedback from the task-specific model to guide the generation of new training…
-
12 Dec 2021 2 repositories listedAt the heart of our approach is a novel strategy to reuse the shared common features in training data so as to synthesize different data instances.
-
18 May 2021 2 repositories listedIn this paper, we propose Contrastive Model Inversion (CMI), where the data diversity is explicitly modeled as an optimizable objective, to alleviate the mode collapse issue.
-
19 Oct 2017 2 repositories listedRecent advances in model compression have provided procedures for compressing large neural networks to a fraction of their original size while retaining most if not all of their accuracy.
-
18 Feb 2025 1 repository listedAs synthetic data becomes increasingly popular in machine learning tasks, numerous methods--without formal differential privacy guarantees--use synthetic data for training.
-
18 Dec 2024 1 repository listedIn the student distillation module, we explore a data inflation strategy to properly utilize a blend of real and synthetic data to train the student network via a classifier-sharing-based feature alignment technique.
-
16 Dec 2024 1 repository listedThen, in the student training phase, we perform an opposite optimization, which adversarially attempts to reduce the distance of samples of the same classes and enlarge the distance of samples of different classes.
-
6 Dec 2024 1 repository listed Syntology official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 1 honoured, 2 violated, 3 with no contract checked; 4 where Syntology's instrument failed) · 1 unverified (of 11 harvested samples)Additionally, they may encounter scalability issues with complex datasets due to inherent two-step information loss: first, during local training (from data to model), and second, when transferring knowledge to the…
-
11 Sep 2024 1 repository listedThe application of machine learning to medical ultrasound videos of the heart, i. e., echocardiography, has recently gained traction with the availability of large public datasets.
-
5 Sep 2024 1 repository listedFurthermore, considering that the source data is either unaccessible or too enormous to store for current generative models, we introduce a new paradigm for their distillation without source data, termed Data-Free…
-
2 Sep 2024 1 repository listed Syntology official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 8 where Syntology's instrument failed) · 4 unverified (of 13 harvested samples) · 13 pointer-only (licence)Federated Class Continual Learning (FCCL) merges the challenges of distributed client learning with the need for seamless adaptation to new classes without forgetting old ones.
-
12 Jun 2024 1 repository listedIn this line of research, existing methods typically follow an inversion-and-distillation paradigm in which a generative adversarial network on-the-fly trained with the guidance of the pre-trained teacher network is…
-
18 Apr 2024 1 repository listedOur approach utilizes an adversarial distillation framework with attention generator, mixed high-order attention distillation, and semantic feature contrast learning.
-
11 Mar 2024 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 2 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 0 violated, 3 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 7 harvested samples)However, simply adopting models derived from DFKD for real-world applications suffers significant performance degradation, due to the discrepancy between teachers' training data and real-world scenarios (student domain).
-
18 Feb 2024 1 repository listedIn this paper, we propose the teacher-agnostic data-free knowledge distillation (TA-DFKD) method, with the goal of more robust and stable performance regardless of teacher models.
-
24 Oct 2023 1 repository listedThe current state-of-the-art decentralized learning algorithms mostly assume the data distribution to be Independent and Identically Distributed (IID).
-
30 Sep 2023 1 repository listed Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 3 where Syntology's instrument failed) · 3 unverified (of 10 harvested samples) · 10 pointer-only (licence)In this paper, we propose a novel Noisy Layer Generation method (NAYER) which relocates the random source from the input to a noisy layer and utilizes the meaningful constant label-text embedding (LTE) as the input.
-
23 Aug 2023 1 repository listedIn this paper, we propose Prompt2Model, a general-purpose method that takes a natural language task description like the prompts provided to LLMs, and uses it to train a special-purpose model that is conducive to…
-
10 Jul 2023 1 repository listedExisting works generally synthesize data from the pre-trained teacher model to replace the original training data for student learning.
-
4 Jun 2023 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 7 harvested samples) · 3 pointer-only (licence)Data-free knowledge distillation (KD) helps transfer knowledge from a pre-trained model (known as the teacher model) to a smaller model (known as the student model) without access to the original training data used for…
-
22 May 2023 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)Diffusion models have recently achieved astonishing performance in generating high-fidelity photo-realistic images.
-
11 Jan 2023 1 repository listedKnowledge distillation is the technique of compressing a larger neural network, known as the teacher, into a smaller neural network, known as the student, while still trying to maintain the performance of the larger…
-
1 Jan 2023 1 repository listedTherefore, we propose mSARC to assure the student network can imitate not only the logit output but also the spatial activation region of the teacher network in order to alleviate the influence of unwanted noises in…
-
29 Aug 2022 1 repository listedBesides, CuDFKD adapts the generation target dynamically according to the status of student model.
-
23 Jul 2022 1 repository listedThe key idea in FedKF is to let the server return the global knowledge to be fused with the local knowledge in each training round so that the local model can be regularized towards the global optima.
-
24 May 2022 1 repository listedTo tackle this challenge, we propose a framework termed collaborative data-free knowledge distillation via multi-level feature sharing (CDFKD-MFS), which consists of a multi-header student module, an asymmetric…
-
17 Mar 2022 1 repository listed Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample)Instead, we propose a data-free knowledge distillation method to fine-tune the global model in the server (FedFTG), which relieves the issue of direct model aggregation.
-
9 Jan 2022 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
9 Oct 2021 1 repository listed Syntology official (archive's flag): 15 ran · 15 ran (of which 5 constructed an object rather than computing a result; 7 with no instrument failure: 2 honoured, 0 violated, 5 with no contract checked; 8 where Syntology's instrument failed) · 5 unverified (of 20 harvested samples)
-
28 Feb 2021 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
7 Nov 2020 1 repository listed
-
2 Nov 2020 1 repository listed
-
6 May 2020 1 repository listed
-
1 Dec 2019 1 repository listed
-
Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environments26 May 2025 0 repositories listed
-
CAE-DFKD: Bridging the Transferability Gap in Data-Free Knowledge Distillation30 Apr 2025 0 repositories listed
-
Adversarial Curriculum Graph-Free Knowledge Distillation for Graph Neural Networks1 Apr 2025 0 repositories listed
-
HFedCKD: Toward Robust Heterogeneous Federated Learning via Data-free Knowledge Distillation and Two-way Contrast9 Mar 2025 0 repositories listed
-
Knowledge Migration Framework for Smart Contract Vulnerability Detection15 Dec 2024 0 repositories listed
-
Large-Scale Data-Free Knowledge Distillation for ImageNet via Multi-Resolution Data Generation26 Nov 2024 0 repositories listed
-
Towards Lifelong Few-Shot Customization of Text-to-Image Diffusion8 Nov 2024 0 repositories listed
-
Towards Effective Data-Free Knowledge Distillation via Diverse Diffusion Augmentation23 Oct 2024 0 repositories listed
-
Leveraging Foundation Models for Efficient Federated Learning in Resource-restricted Edge Networks14 Sep 2024 0 repositories listed
-
Distilling Vision-Language Foundation Models: A Data-Free Approach via Prompt Diversification21 Jul 2024 0 repositories listed
-
Mind the Gap Between Synthetic and Real: Utilizing Transfer Learning to Probe the Boundaries of Stable Diffusion Generated Data6 May 2024 0 repositories listed
-
FedTAD: Topology-aware Data-free Knowledge Distillation for Subgraph Federated Learning22 Apr 2024 0 repositories listed
-
De-confounded Data-free Knowledge Distillation for Handling Distribution Shifts28 Mar 2024 0 repositories listed
-
28 Mar 2024 0 repositories listed
-
FedD2S: Personalized Data-Free Federated Knowledge Distillation16 Feb 2024 0 repositories listed
-
Data-Free Distillation of Language Model by Text-to-Text Transfer3 Nov 2023 0 repositories listed
-
Data-Free Knowledge Distillation Using Adversarially Perturbed OpenGL Shader Images20 Oct 2023 0 repositories listed
-
DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learning24 Sep 2023 0 repositories listed
-
Sampling to Distill: Knowledge Transfer from Open-World Data31 Jul 2023 0 repositories listed
-
Mitigating Cross-client GANs-based Attack in Federated Learning25 Jul 2023 0 repositories listed
-
Distribution Shift Matters for Knowledge Distillation with Webly Collected Images21 Jul 2023 0 repositories listed
-
Feature-Rich Audio Model Inversion for Data-Free Knowledge Distillation Towards General Sound Classification14 Mar 2023 0 repositories listed
-
NIFF: Alleviating Forgetting in Generalized Few-Shot Object Detection via Neural Instance Feature Forging9 Mar 2023 0 repositories listed
-
Learning to Retain while Acquiring: Combating Distribution-Shift in Adversarial Data-Free Knowledge Distillation28 Feb 2023 0 repositories listed
-
Explicit and Implicit Knowledge Distillation via Unlabeled Data17 Feb 2023 0 repositories listed
-
Completely Heterogeneous Federated Learning28 Oct 2022 0 repositories listed
-
Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation21 Sep 2022 0 repositories listed
-
Dense Depth Distillation with Out-of-Distribution Simulated Images26 Aug 2022 0 repositories listed
-
Prompting to Distill: Boosting Data-Free Knowledge Distillation via Reinforced Prompt16 May 2022 0 repositories listed
-
FedDTG:Federated Data-Free Knowledge Distillation via Three-Player Generative Adversarial Networks10 Jan 2022 0 repositories listed
-
Conditional Generative Data-free Knowledge Distillation31 Dec 2021 0 repositories listed
-
Data-Free Knowledge Transfer: A Survey31 Dec 2021 0 repositories listed
-
Data-Free Knowledge Distillation for Image Super-Resolution19 Jun 2021 0 repositories listed
-
Dual Discriminator Adversarial Distillation for Data-free Model Compression12 Apr 2021 0 repositories listed
-
Data-Free Knowledge Distillation with Soft Targeted Transfer Set Synthesis10 Apr 2021 0 repositories listed
-
Generative Adversarial Simulator23 Nov 2020 0 repositories listed
-
Effectiveness of Arbitrary Transfer Sets for Data-free Knowledge Distillation18 Nov 2020 0 repositories listed
-
DeGAN : Data-Enriching GAN for Retrieving Representative Samples from a Trained Classifier27 Dec 2019 0 repositories listed
Syntology lines on 11 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections