Data-free Knowledge Distillation

Benchmarks archive 2025-07-28

2 leaderboard tables shown for this task, 2 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

3 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

1 subtask in the archive's task tree.

Parent tasks archive 2025-07-28

Most implemented papers archive 2025-07-28

30 shown of 37 papers with code (75 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• Data-Free Knowledge Distillation for Heterogeneous Federated Learning
  20 May 2021 4 repositories listed
  Federated Learning (FL) is a decentralized machine-learning paradigm, in which a global server iteratively averages the model parameters of local users without accessing their data.
• ZeroGen: Efficient Zero-shot Learning via Dataset Generation
  16 Feb 2022 3 repositories listed Syntology ran 3 of 7 samples · 4 unverified · 7 pointer-only (licence)
  There is a growing interest in dataset generation recently due to the superior generative capacity of large pre-trained language models (PLMs).
• DAD++: Improved Data-free Test Time Adversarial Defense
  10 Sep 2023 2 repositories listed
  With the increasing deployment of deep neural networks in safety-critical applications such as self-driving cars, medical imaging, anomaly detection, etc., adversarial robustness has become a crucial concern in the…
• ProGen: Progressive Zero-shot Dataset Generation via In-context Feedback
  22 Oct 2022 2 repositories listed
  To improve the quality of dataset synthesis, we propose a progressive zero-shot dataset generation framework, ProGen, which leverages the feedback from the task-specific model to guide the generation of new training…
• Up to 100× Faster Data-free Knowledge Distillation
  12 Dec 2021 2 repositories listed
  At the heart of our approach is a novel strategy to reuse the shared common features in training data so as to synthesize different data instances.
• Contrastive Model Inversion for Data-Free Knowledge Distillation
  18 May 2021 2 repositories listed
  In this paper, we propose Contrastive Model Inversion~(CMI), where the data diversity is explicitly modeled as an optimizable objective, to alleviate the mode collapse issue.
• Data-Free Knowledge Distillation for Deep Neural Networks
  19 Oct 2017 2 repositories listed
  Recent advances in model compression have provided procedures for compressing large neural networks to a fraction of their original size while retaining most if not all of their accuracy.
• Does Training with Synthetic Data Truly Protect Privacy?
  18 Feb 2025 1 repository listed
  As synthetic data becomes increasingly popular in machine learning tasks, numerous methods--without formal differential privacy guarantees--use synthetic data for training.
• Hybrid Data-Free Knowledge Distillation
  18 Dec 2024 1 repository listed
  In the student distillation module, we explore a data inflation strategy to properly utilize a blend of real and synthetic data to train the student network via a classifier-sharing-based feature alignment technique.
• Relation-Guided Adversarial Learning for Data-free Knowledge Transfer
  16 Dec 2024 1 repository listed
  Then, in the student training phase, we perform an opposite optimization, which adversarially attempts to reduce the distance of samples of the same classes and enlarge the distance of samples of different classes.
• One-shot Federated Learning via Synthetic Distiller-Distillate Communication
  6 Dec 2024 1 repository listed Syntology ran 3 of 11 samples · 8 unverified
  Additionally, they may encounter scalability issues with complex datasets due to inherent two-step information loss: first, during local training (from data to model), and second, when transferring knowledge to the…
• EchoDFKD: Data-Free Knowledge Distillation for Cardiac Ultrasound Segmentation using Synthetic Data
  11 Sep 2024 1 repository listed
  The application of machine learning to medical ultrasound videos of the heart, i.
• DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecture
  5 Sep 2024 1 repository listed
  Furthermore, considering that the source data is either unaccessible or too enormous to store for current generative models, we introduce a new paradigm for their distillation without source data, termed Data-Free…
• Diffusion-Driven Data Replay: A Novel Approach to Combat Forgetting in Federated Class Continual Learning
  2 Sep 2024 1 repository listed Syntology ran 8 of 13 samples · 5 unverified · 13 pointer-only (licence)
  Federated Class Continual Learning (FCCL) merges the challenges of distributed client learning with the need for seamless adaptation to new classes without forgetting old ones.
• Small Scale Data-Free Knowledge Distillation
  12 Jun 2024 1 repository listed
  In this line of research, existing methods typically follow an inversion-and-distillation paradigm in which a generative adversarial network on-the-fly trained with the guidance of the pre-trained teacher network is…
• Data-free Knowledge Distillation for Fine-grained Visual Categorization
  18 Apr 2024 1 repository listed
  Our approach utilizes an adversarial distillation framework with attention generator, mixed high-order attention distillation, and semantic feature contrast learning.
• AuG-KD: Anchor-Based Mixup Generation for Out-of-Domain Knowledge Distillation
  11 Mar 2024 1 repository listed Syntology ran 6 of 7 samples · 1 unverified
  However, simply adopting models derived from DFKD for real-world applications suffers significant performance degradation, due to the discrepancy between teachers' training data and real-world scenarios (student domain).
• Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation
  18 Feb 2024 1 repository listed
  In this paper, we propose the teacher-agnostic data-free knowledge distillation (TA-DFKD) method, with the goal of more robust and stable performance regardless of teacher models.
• Cross-feature Contrastive Loss for Decentralized Deep Learning on Heterogeneous Data
  24 Oct 2023 1 repository listed
  The current state-of-the-art decentralized learning algorithms mostly assume the data distribution to be Independent and Identically Distributed (IID).
• NAYER: Noisy Layer Data Generation for Efficient and Effective Data-free Knowledge Distillation
  30 Sep 2023 1 repository listed Syntology ran 7 of 10 samples · 3 unverified · 10 pointer-only (licence)
  In this paper, we propose a novel Noisy Layer Generation method (NAYER) which relocates the random source from the input to a noisy layer and utilizes the meaningful constant label-text embedding (LTE) as the input.
• Prompt2Model: Generating Deployable Models from Natural Language Instructions
  23 Aug 2023 1 repository listed
  In this paper, we propose Prompt2Model, a general-purpose method that takes a natural language task description like the prompts provided to LLMs, and uses it to train a special-purpose model that is conducive to…
• Customizing Synthetic Data for Data-Free Student Learning
  10 Jul 2023 1 repository listed
  Existing works generally synthesize data from the pre-trained teacher model to replace the original training data for student learning.
• Revisiting Data-Free Knowledge Distillation with Poisoned Teachers
  4 Jun 2023 1 repository listed Syntology ran 2 of 7 samples · 5 unverified
  Data-free knowledge distillation (KD) helps transfer knowledge from a pre-trained model (known as the teacher model) to a smaller model (known as the student model) without access to the original training data used for…
• Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?
  22 May 2023 1 repository listed Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
  Diffusion models have recently achieved astonishing performance in generating high-fidelity photo-realistic images.
• Synthetic data generation method for data-free knowledge distillation in regression neural networks
  11 Jan 2023 1 repository listed
  Knowledge distillation is the technique of compressing a larger neural network, known as the teacher, into a smaller neural network, known as the student, while still trying to maintain the performance of the larger…
• Data-Free Knowledge Distillation via Feature Exchange and Activation Region Constraint
  1 Jan 2023 1 repository listed
  Therefore, we propose mSARC to assure the student network can imitate not only the logit output but also the spatial activation region of the teacher network in order to alleviate the influence of unwanted noises in…
• Dynamic Data-Free Knowledge Distillation by Easy-to-Hard Learning Strategy
  29 Aug 2022 1 repository listed
  Besides, CuDFKD adapts the generation target dynamically according to the status of student model.
• Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and Fusion
  23 Jul 2022 1 repository listed
  The key idea in FedKF is to let the server return the global knowledge to be fused with the local knowledge in each training round so that the local model can be regularized towards the global optima.
• CDFKD-MFS: Collaborative Data-free Knowledge Distillation via Multi-level Feature Sharing
  24 May 2022 1 repository listed
  To tackle this challenge, we propose a framework termed collaborative data-free knowledge distillation via multi-level feature sharing (CDFKD-MFS), which consists of a multi-header student module, an asymmetric…
• Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning
  17 Mar 2022 1 repository listed Syntology ran 0 of 1 samples · 1 unverified
  Instead, we propose a data-free knowledge distillation method to fine-tune the global model in the server (FedFTG), which relieves the issue of direct model aggregation.

Syntology lines on 8 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.