Self-Knowledge Distillation

Benchmarks archive 2025-07-28

No benchmark for this task in the archive.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

No dataset record in the archive lists this task.

Subtasks archive 2025-07-28

No subtask under this task in the archive's task tree.

Parent tasks archive 2025-07-28

Most implemented papers archive 2025-07-28

30 shown of 36 papers with code (68 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• ProSelfLC: Progressive Self Label Correction for Training Robust Deep Neural Networks
  7 May 2020 5 repositories listed Syntology ran 1 of 11 samples · 10 unverified
  Keywords: entropy minimisation, maximum entropy, confidence penalty, self knowledge distillation, label correction, label noise, semi-supervised learning, output regularisation
• BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
  5 Feb 2024 4 repositories listed Syntology ran 3 of 14 samples · 11 unverified
  It can simultaneously perform the three common retrieval functionalities of embedding model: dense retrieval, multi-vector retrieval, and sparse retrieval, which provides a unified model foundation for real-world IR…
• Preservation of the Global Knowledge by Not-True Distillation in Federated Learning
  6 Jun 2021 3 repositories listed Syntology ran 4 of 9 samples · 5 unverified · 5 pointer-only (licence)
  In federated learning, a strong global model is collaboratively learned by aggregating clients' locally trained models.
• FedSOL: Stabilized Orthogonal Learning with Proximal Restrictions in Federated Learning
  24 Aug 2023 2 repositories listed Syntology ran 10 of 11 samples · 1 unverified
  FedSOL is designed to identify gradients of local objectives that are inherently orthogonal to directions affecting the proximal objective.
• Revisiting Knowledge Distillation via Label Smoothing Regularization
  25 Sep 2019 2 repositories listed Syntology ran 0 of 13 samples · 13 unverified
  Without any extra computation cost, Tf-KD achieves up to 0.
• Tackling Data Heterogeneity in Federated Learning through Knowledge Distillation with Inequitable Aggregation
  25 Jun 2025 1 repository listed
  Federated learning aims to train a global model in a distributed environment that is close to the performance of centralized training.
• Efficient Lung Ultrasound Severity Scoring Using Dedicated Feature Extractor
  21 Jan 2025 1 repository listed
  With the advent of the COVID-19 pandemic, ultrasound imaging has emerged as a promising technique for COVID-19 detection, due to its non-invasive nature, affordability, and portability.
• Frequency-Guided Masking for Enhanced Vision Self-Supervised Learning
  16 Sep 2024 1 repository listed
  Second, we employ a two-branch framework empowered by knowledge distillation, enabling the model to take both the filtered and original images as input, largely reducing the burden of downstream tasks.
• SalNAS: Efficient Saliency-prediction Neural Architecture Search with self-knowledge distillation
  29 Jul 2024 1 repository listed
  Recent advancements in deep convolutional neural networks have significantly improved the performance of saliency prediction.
• Towards A Generalizable Pathology Foundation Model via Unified Knowledge Distillation
  26 Jul 2024 1 repository listed Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
  Foundation models pretrained on large-scale datasets are revolutionizing the field of computational pathology (CPath).
• Three-Stream Temporal-Shift Attention Network Based on Self-Knowledge Distillation for Micro-Expression Recognition
  25 Jun 2024 1 repository listed
  Secondly, we employ efficient channel attention modules in the local-spatial stream to make the network focus on facial regions that are highly relevant to micro-expressions.
• Guiding Frame-Level CTC Alignments Using Self-knowledge Distillation
  12 Jun 2024 1 repository listed
  Transformer encoder with connectionist temporal classification (CTC) framework is widely used for automatic speech recognition (ASR).
• Vision-Language Meets the Skeleton: Progressively Distillation with Cross-Modal Knowledge for 3D Action Representation Learning
  31 May 2024 1 repository listed
  To address these challenges, we introduce a novel skeleton-based training framework (C²VL) based on Cross-modal Contrastive learning that uses the progressive distillation to learn task-agnostic human skeleton action…
• CrossMatch: Enhance Semi-Supervised Medical Image Segmentation with Perturbation Strategies and Knowledge Distillation
  1 May 2024 1 repository listed
  Semi-supervised learning for medical image segmentation presents a unique challenge of efficiently using limited labeled data while leveraging abundant unlabeled data.
• Distilled Gradual Pruning with Pruned Fine-tuning
  15 Feb 2024 1 repository listed
  Neural Networks (NNs) have been driving machine learning progress in recent years, but their larger models present challenges in resource-limited environments.
• Deep Clustering with Diffused Sampling and Hardness-aware Self-distillation
  25 Jan 2024 1 repository listed
  Deep clustering has gained significant attention due to its capability in learning clustering-friendly representations without labeled data.
• Combining inherent knowledge of vision-language models with unsupervised domain adaptation through strong-weak guidance
  7 Dec 2023 1 repository listed
  The weak guidance is implemented as a knowledge distillation loss with (shifted) zero-shot predictions.
• Promoting Generalized Cross-lingual Question Answering in Few-resource Scenarios via Self-knowledge Distillation
  29 Sep 2023 1 repository listed
  Our approach seeks to enhance cross-lingual QA transfer using a high-performing multilingual model trained on a large-scale dataset, complemented by a few thousand aligned QA examples across languages.
• Eyelid’s Intrinsic Motion-aware Feature Learning for Real-time Eyeblink Detection in the Wild
  3 Aug 2023 1 repository listed
  The experiments verify that, our proposition is significantly superior to the state-of-the-art ones, and with real-time running efficiency.
• Effective Whole-body Pose Estimation with Two-stages Distillation
  29 Jul 2023 1 repository listed Syntology ran 2 of 8 samples · 6 unverified
  Different from the previous self-knowledge distillation, this stage finetunes the student's head with only 20% training time as a plug-and-play training strategy.
• Robust Spatiotemporal Traffic Forecasting with Reinforced Dynamic Adversarial Training
  25 Jun 2023 1 repository listed
  Therefore, improving the adversarial robustness of these models is crucial for ITS.
• Incorporating Graph Information in Transformer-based AMR Parsing
  23 Jun 2023 1 repository listed
  Abstract Meaning Representation (AMR) is a Semantic Parsing formalism that aims at providing a semantic graph abstraction representing a given text.
• Lightweight Self-Knowledge Distillation with Multi-source Information Fusion
  16 May 2023 1 repository listed
  Specifically, we introduce a Distillation with Reverse Guidance (DRG) method that considers different levels of information extracted by the model, including edge, shape, and detail of the input data, to construct a…
• From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels
  23 Mar 2023 1 repository listed
  We decompose the KD loss and find the non-target loss from it forces the student's non-target logits to match the teacher's, but the sum of the two non-target logits is different, preventing them from being identical.
• DualFair: Fair Representation Learning at Both Group and Individual Levels via Contrastive Self-supervision
  15 Mar 2023 1 repository listed Syntology ran 2 of 3 samples · 1 unverified
  Algorithmic fairness has become an important machine learning problem, especially for mission-critical Web applications.
• Graph-based Knowledge Distillation: A survey and experimental evaluation
  27 Feb 2023 1 repository listed
  It then provides a comprehensive summary of three types of Graph-based Knowledge Distillation methods, namely Graph-based Knowledge Distillation for deep neural networks (DKD), Graph-based Knowledge Distillation for…
• Multimodality Multi-Lead ECG Arrhythmia Classification using Self-Supervised Learning
  30 Sep 2022 1 repository listed
  Electrocardiogram (ECG) signal is one of the most effective sources of information mainly employed for the diagnosis and prediction of cardiovascular diseases (CVDs) connected with the abnormalities in heart rhythm.
• RAIN: RegulArization on Input and Network for Black-Box Domain Adaptation
  22 Aug 2022 1 repository listed
  For the input-level, we design a new data augmentation technique as Phase MixUp, which highlights task-relevant objects in the interpolations, thus enhancing input-level regularization and class consistency for target…
• MixSKD: Self-Knowledge Distillation from Mixup for Image Recognition
  11 Aug 2022 1 repository listed Syntology ran 3 of 4 samples · 1 unverified · 4 pointer-only (licence)
  MixSKD mutually distills feature maps and probability distributions between the random pair of original images and their mixup images in a meaningful way.
• Improving Sequential Recommendations via Bidirectional Temporal Data Augmentation with Pre-training
  13 Dec 2021 1 repository listed
  Our approach leverages bidirectional temporal augmentation and knowledge-enhanced fine-tuning to synthesize authentic pseudo-prior items that retain user preferences and capture deeper item semantic correlations, thus…

Syntology lines on 9 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.