Methods › General › Data Parallel Methods › Gradient Sparsification
Gradient Sparsification
Introduced by Jianqiao Wangni et al. in Gradient Sparsification for Communication-Efficient Distributed Optimization
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Gradient Sparsification is a technique for distributed training that sparsifies stochastic gradients to reduce the communication cost, with minor increase in the number of iterations. The key idea behind our sparsification technique is to drop some coordinates of the stochastic gradient and appropriately amplify the remaining coordinates to ensure the unbiasedness of the sparsified stochastic gradient. The sparsification approach can significantly reduce the coding length of the stochastic gradient and only slightly increase the variance of the stochastic gradient.
Papers archive 2025-07-28
30 shown of 38, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
Mobility-Aware Asynchronous Federated Learning with Dynamic Sparsification 8 Jun 2025 · 0 repositories · arXiv:2506.07328
-
Dynamic Gradient Sparsification Training for Few-Shot Fine-tuning of CT Lymph Node Segmentation Foundation Model 2 Mar 2025 · 1 repository · arXiv:2503.00748
-
Sparse Incremental Aggregation in Satellite Federated Learning 20 Jan 2025 · 0 repositories · arXiv:2501.11385
-
Regularized Top-k: A Bayesian Framework for Gradient Sparsification 10 Jan 2025 · 0 repositories · arXiv:2501.05633
-
DQRM: Deep Quantized Recommendation Models 26 Oct 2024 · 1 repository · arXiv:2410.20046
-
Age-of-Gradient Updates for Federated Learning over Random Access Channels 15 Oct 2024 · 0 repositories · arXiv:2410.11986
-
Novel Gradient Sparsification Algorithm via Bayesian Inference 23 Sep 2024 · 0 repositories · arXiv:2409.14893
-
Preserving Near-Optimal Gradient Sparsification Cost for Scalable Distributed Deep Learning 21 Feb 2024 · 1 repository · arXiv:2402.13781
-
JointSQ: Joint Sparsification-Quantization for Distributed Learning 1 Jan 2024 · 1 repository
-
RS-DGC: Exploring Neighborhood Statistics for Dynamic Gradient Compression on Remote Sensing Image Interpretation 29 Dec 2023 · 0 repositories · arXiv:2312.17530
-
MiCRO: Near-Zero Cost Gradient Sparsification for Scaling and Accelerating Distributed DNN Training 2 Oct 2023 · 1 repository · arXiv:2310.00967
-
Gradient Sparsification For Masked Fine-Tuning of Transformers 19 Jul 2023 · 0 repositories · arXiv:2307.10098
-
DEFT: Exploiting Gradient Norm Difference between Model Layers for Scalable Gradient Sparsification 7 Jul 2023 · 1 repository · arXiv:2307.03500
-
Gradient Sparsification for Efficient Wireless Federated Learning with Differential Privacy 9 Apr 2023 · 0 repositories · arXiv:2304.04164
-
Efficient and Secure Federated Learning for Financial Applications 15 Mar 2023 · 0 repositories · arXiv:2303.08355
-
On the Interaction Between Differential Privacy and Gradient Compression in Deep Learning 1 Nov 2022 · 0 repositories · arXiv:2211.00734
-
Downlink Compression Improves TopK Sparsification 30 Sep 2022 · 0 repositories · arXiv:2209.15203
-
Empirical Analysis on Top-k Gradient Sparsification for Distributed Deep Learning in a Supercomputing Environment 18 Sep 2022 · 0 repositories · arXiv:2209.08497
-
Near-Optimal Sparse Allreduce for Distributed Deep Learning 19 Jan 2022 · 1 repository · arXiv:2201.07598
-
Sparsified Secure Aggregation for Privacy-Preserving Federated Learning 23 Dec 2021 · 0 repositories · arXiv:2112.12872
-
Communication-Efficient Federated Learning via Quantized Compressed Sensing 30 Nov 2021 · 0 repositories · arXiv:2111.15071
-
Federated Dynamic Neural Network for Deep MIMO Detection 24 Nov 2021 · 0 repositories · arXiv:2111.12260
-
Gradient Sparsification For \emph{Masked Fine-Tuning} of Transformers 16 Nov 2021 · 0 repositories
-
Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradients 14 Feb 2021 · 0 repositories · arXiv:2102.07053
-
Large-Scale Training System for 100-Million Classification at Alibaba 9 Feb 2021 · 0 repositories · arXiv:2102.06025
-
Gradient Sparsification Can Improve Performance of Differentially-Private Convex Machine Learning 30 Nov 2020 · 0 repositories · arXiv:2011.14572
-
A Reputation Mechanism Is All You Need: Collaborative Fairness and Adversarial Robustness in Federated Learning 20 Nov 2020 · 2 repositories · arXiv:2011.10464
-
FLAME: Differentially Private Federated Learning in the Shuffle Model 17 Sep 2020 · 1 repository · arXiv:2009.08063Syntology ran 0 of 1 samples · 1 unverified
-
rTop-k: A Statistical Estimation Approach to Distributed SGD 21 May 2020 · 0 repositories · arXiv:2005.10761
-
Adaptive Gradient Sparsification for Efficient Federated Learning: An Online Learning Approach 14 Jan 2020 · 0 repositories · arXiv:2001.04756
Tasks archive 2025-07-28
20 shown of 28 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections