{"url":"/method/gradient-clipping","slug":"gradient-clipping","name":"Gradient Clipping","full_name":"Gradient Clipping","full_name_withheld":false,"description_markdown":"One difficulty that arises with optimization of deep neural networks is that large parameter gradients can lead an [SGD](https://paperswithcode.com/method/sgd) optimizer to update the parameters strongly into a region where the loss function is much greater, effectively undoing much of the work that was needed to get to the current solution.\r\n\r\n**Gradient Clipping** clips the size of the gradients to ensure optimization performs more reasonably near sharp areas of the loss surface. It can be performed in a number of ways. One option is to simply clip the parameter gradient element-wise before a parameter update. Another option is to clip the norm ||$\\textbf{g}$|| of the gradient $\\textbf{g}$ before a parameter update:\r\n\r\n$$\\text{ if } ||\\textbf{g}||  > v \\text{ then } \\textbf{g} \\leftarrow \\frac{\\textbf{g}{v}}{||\\textbf{g}||}$$\r\n\r\nwhere $v$ is a norm threshold.\r\n\r\nSource: Deep Learning, Goodfellow et al\r\n\r\nImage Source: [Pascanu et al](https://arxiv.org/pdf/1211.5063.pdf)","description_state":"present","introduced_year":null,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":null,"title":null,"url_on_a_paper_host":false},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Optimization","url":"/methods/category/optimization","pwc_aliases":[]}],"n_papers_tagged":167,"archive_num_papers":167,"papers_newest_first":[{"paper":"/paper/2506-08347","title":"Differentially Private Relational Learning with Entity-level Privacy Guarantees","date":"2025-06-10","arxiv_id":"2506.08347","n_code_links":1,"syntology":null},{"paper":null,"title":"GeoClip: Geometry-Aware Clipping for Differentially Private SGD","date":"2025-06-06","arxiv_id":"2506.06549","n_code_links":0,"syntology":null},{"paper":null,"title":"GCFL: A Gradient Correction-based Federated Learning Framework for Privacy-preserving CPSS","date":"2025-06-04","arxiv_id":"2506.03618","n_code_links":0,"syntology":null},{"paper":"/paper/sample-and-computationally-efficient-1","title":"Sample and Computationally Efficient Continuous-Time Reinforcement Learning with General Function Approximation","date":"2025-05-20","arxiv_id":"2505.14821","n_code_links":1,"syntology":null},{"paper":"/paper/a-training-framework-for-optimal-and-stable","title":"A Training Framework for Optimal and Stable Training of Polynomial Neural Networks","date":"2025-05-16","arxiv_id":"2505.11589","n_code_links":1,"syntology":null},{"paper":null,"title":"Dyn-D$^2$P: Dynamic Differentially Private Decentralized Learning with Provable Utility Guarantee","date":"2025-05-10","arxiv_id":"2505.06651","n_code_links":0,"syntology":null},{"paper":null,"title":"Can Local Representation Alignment RNNs Solve Temporal Tasks?","date":"2025-04-18","arxiv_id":"2504.13531","n_code_links":0,"syntology":null},{"paper":null,"title":"Technical Report: Full Version of Analyzing and Optimizing Perturbation of DP-SGD Geometrically","date":"2025-04-08","arxiv_id":"2504.05618","n_code_links":0,"syntology":null},{"paper":"/paper/zclip-adaptive-spike-mitigation-for-llm-pre","title":"ZClip: Adaptive Spike Mitigation for LLM Pre-Training","date":"2025-04-03","arxiv_id":"2504.02507","n_code_links":1,"syntology":{"ran":0,"of":1,"unverified":1,"pointer_only":0}},{"paper":"/paper/world-model-agents-with-change-based","title":"World Model Agents with Change-Based Intrinsic Motivation","date":"2025-03-26","arxiv_id":"2503.21047","n_code_links":1,"syntology":null},{"paper":null,"title":"Tractable Representations for Convergent Approximation of Distributional HJB Equations","date":"2025-03-07","arxiv_id":"2503.05563","n_code_links":0,"syntology":null},{"paper":null,"title":"AdaGC: Improving Training Stability for Large Language Model Pretraining","date":"2025-02-16","arxiv_id":"2502.11034","n_code_links":0,"syntology":null},{"paper":null,"title":"Local Differential Privacy is Not Enough: A Sample Reconstruction Attack against Federated Learning with Local Differential Privacy","date":"2025-02-12","arxiv_id":"2502.08151","n_code_links":0,"syntology":null},{"paper":"/paper/mitigating-unintended-memorization-with-lora","title":"Mitigating Unintended Memorization with LoRA in Federated Learning for LLMs","date":"2025-02-07","arxiv_id":"2502.05087","n_code_links":1,"syntology":null},{"paper":null,"title":"BMG-Q: Localized Bipartite Match Graph Attention Q-Learning for Ride-Pooling Order Dispatch","date":"2025-01-23","arxiv_id":"2501.13448","n_code_links":0,"syntology":null},{"paper":null,"title":"Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions","date":"2025-01-08","arxiv_id":"2501.04437","n_code_links":0,"syntology":null},{"paper":null,"title":"On the Convergence of DP-SGD with Adaptive Clipping","date":"2024-12-27","arxiv_id":"2412.19916","n_code_links":0,"syntology":null},{"paper":"/paper/optimized-gradient-clipping-for-noisy-label","title":"Optimized Gradient Clipping for Noisy Label Learning","date":"2024-12-12","arxiv_id":"2412.08941","n_code_links":1,"syntology":null},{"paper":null,"title":"A Combined Encoder and Transformer Approach for Coherent and High-Quality Text Generation","date":"2024-11-19","arxiv_id":"2411.12157","n_code_links":0,"syntology":null},{"paper":null,"title":"Gradient Normalization Provably Benefits Nonconvex SGD under Heavy-Tailed Noise","date":"2024-10-21","arxiv_id":"2410.16561","n_code_links":0,"syntology":null},{"paper":null,"title":"From Gradient Clipping to Normalization for Heavy Tailed SGD","date":"2024-10-17","arxiv_id":"2410.13849","n_code_links":0,"syntology":null},{"paper":null,"title":"Adaptive Gradient Normalization and Independent Sampling for (Stochastic) Generalized-Smooth Optimization","date":"2024-10-17","arxiv_id":"2410.14054","n_code_links":0,"syntology":null},{"paper":null,"title":"Private and Communication-Efficient Federated Learning based on Differentially Private Sketches","date":"2024-10-08","arxiv_id":"2410.05733","n_code_links":0,"syntology":null},{"paper":null,"title":"Methods for Convex $(L_0,L_1)$-Smooth Optimization: Clipping, Acceleration, and Adaptivity","date":"2024-09-23","arxiv_id":"2409.14989","n_code_links":0,"syntology":null},{"paper":null,"title":"Robust Federated Learning Over the Air: Combating Heavy-Tailed Noise with Median Anchored Clipping","date":"2024-09-23","arxiv_id":"2409.15100","n_code_links":0,"syntology":null},{"paper":null,"title":"Convergence of Distributed Adaptive Optimization with Local Updates","date":"2024-09-20","arxiv_id":"2409.13155","n_code_links":0,"syntology":null},{"paper":"/paper/denomamba-a-fused-state-space-model-for-low","title":"DenoMamba: A fused state-space model for low-dose CT denoising","date":"2024-09-19","arxiv_id":"2409.13094","n_code_links":1,"syntology":null},{"paper":"/paper/causal-temporal-representation-learning-with","title":"Causal Temporal Representation Learning with Nonstationary Sparse Transition","date":"2024-09-05","arxiv_id":"2409.03142","n_code_links":1,"syntology":{"ran":7,"of":9,"unverified":2,"pointer_only":9}},{"paper":"/paper/machine-learning-models-for-daily-rainfall","title":"Machine learning models for daily rainfall forecasting in Northern Tropical Africa using tropical wave predictors","date":"2024-08-29","arxiv_id":"2408.16349","n_code_links":1,"syntology":null},{"paper":null,"title":"Revisit Micro-batch Clipping: Adaptive Data Pruning via Gradient Manipulation","date":"2024-08-29","arxiv_id":"2408.16204","n_code_links":0,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/federated-learning","name":"Federated Learning","papers":12},{"task":"/task/language-modelling","name":"Language Modelling","papers":10},{"task":"/task/reinforcement-learning","name":"Reinforcement Learning","papers":10},{"task":"/task/image-classification","name":"Image Classification","papers":9},{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":9},{"task":"/task/reinforcement-learning-2","name":"reinforcement-learning","papers":9},{"task":"/task/deep-learning","name":"Deep Learning","papers":8},{"task":"/task/language-modeling","name":"Language Modeling","papers":7},{"task":null,"name":"GPU","papers":6},{"task":"/task/privacy-preserving","name":"Privacy Preserving","papers":6},{"task":"/task/representation-learning","name":"Representation Learning","papers":6},{"task":"/task/image-classification","name":"image-classification","papers":6},{"task":"/task/stochastic-optimization","name":"Stochastic Optimization","papers":5},{"task":"/task/text-generation","name":"Text Generation","papers":5},{"task":"/task/deep-reinforcement-learning","name":"Deep Reinforcement Learning","papers":3},{"task":"/task/denoising","name":"Denoising","papers":3},{"task":"/task/fairness","name":"Fairness","papers":3},{"task":"/task/classification","name":"General Classification","papers":3},{"task":"/task/lemma","name":"LEMMA","papers":3},{"task":"/task/large-language-model","name":"Large Language Model","papers":3}],"tasks_shown":20,"n_tasks":150,"usage_by_year":[{"year":"2015","papers":1},{"year":"2016","papers":3},{"year":"2017","papers":3},{"year":"2018","papers":2},{"year":"2019","papers":10},{"year":"2020","papers":13},{"year":"2021","papers":21},{"year":"2022","papers":26},{"year":"2023","papers":42},{"year":"2024","papers":30},{"year":"2025","papers":16}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/gradient-clipping"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}