{"url":"/method/adamw","slug":"adamw","name":"AdamW","full_name":"AdamW","full_name_withheld":false,"description_markdown":"**AdamW** is a stochastic optimization method that modifies the typical implementation of weight decay in [Adam](https://paperswithcode.com/method/adam), by decoupling [weight decay](https://paperswithcode.com/method/weight-decay) from the gradient update. To see this, $L\\_{2}$ regularization in Adam is usually implemented with the below modification where $w\\_{t}$ is the rate of the weight decay at time $t$:\r\n\r\n$$ g\\_{t} = \\nabla{f\\left(\\theta\\_{t}\\right)} + w\\_{t}\\theta\\_{t}$$\r\n\r\nwhile AdamW adjusts the weight decay term to appear in the gradient update:\r\n\r\n$$ \\theta\\_{t+1, i} = \\theta\\_{t, i} - \\eta\\left(\\frac{1}{\\sqrt{\\hat{v}\\_{t} + \\epsilon}}\\cdot{\\hat{m}\\_{t}} + w\\_{t, i}\\theta\\_{t, i}\\right), \\forall{t}$$","description_state":"present","introduced_year":null,"introduced_by":{"title":"Decoupled Weight Decay Regularization","paper":"/paper/decoupled-weight-decay-regularization","first_author":"Ilya Loshchilov","n_authors":2,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/decoupled-weight-decay-regularization"},"source":{"url":"http://arxiv.org/abs/1711.05101v3","title":"Decoupled Weight Decay Regularization","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/pytorch/pytorch/blob/b7bda236d18815052378c88081f64935427d7716/torch/optim/adamw.py#L6","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Stochastic Optimization","url":"/methods/category/stochastic-optimization","pwc_aliases":[]}],"n_papers_tagged":206,"archive_num_papers":206,"papers_newest_first":[{"paper":null,"title":"I Know Which LLM Wrote Your Code Last Summer: LLM generated Code Stylometry for Authorship Attribution","date":"2025-06-18","arxiv_id":"2506.17323","n_code_links":0,"syntology":null},{"paper":null,"title":"Improving LoRA with Variational Learning","date":"2025-06-17","arxiv_id":"2506.14280","n_code_links":0,"syntology":null},{"paper":null,"title":"PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective","date":"2025-05-27","arxiv_id":"2505.21799","n_code_links":0,"syntology":null},{"paper":"/paper/adams-momentum-itself-can-be-a-normalizer-for","title":"AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training","date":"2025-05-22","arxiv_id":"2505.16363","n_code_links":1,"syntology":{"ran":0,"of":1,"unverified":1,"pointer_only":0}},{"paper":"/paper/enhancing-abstractive-summarization-of","title":"Enhancing Abstractive Summarization of Scientific Papers Using Structure Information","date":"2025-05-20","arxiv_id":"2505.14179","n_code_links":1,"syntology":null},{"paper":null,"title":"A Physics-Inspired Optimizer: Velocity Regularized Adam","date":"2025-05-19","arxiv_id":"2505.13196","n_code_links":0,"syntology":null},{"paper":null,"title":"Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training","date":"2025-05-19","arxiv_id":"2505.13738","n_code_links":0,"syntology":null},{"paper":null,"title":"On the $O(\\frac{\\sqrt{d}}{K^{1/4}})$ Convergence Rate of AdamW Measured by $\\ell_1$ Norm","date":"2025-05-17","arxiv_id":"2505.11840","n_code_links":0,"syntology":null},{"paper":null,"title":"Variational Visual Question Answering","date":"2025-05-14","arxiv_id":"2505.09591","n_code_links":0,"syntology":null},{"paper":"/paper/practical-efficiency-of-muon-for-pretraining","title":"Practical Efficiency of Muon for Pretraining","date":"2025-05-04","arxiv_id":"2505.02222","n_code_links":1,"syntology":null},{"paper":null,"title":"CacheFormer: High Attention-Based Segment Caching","date":"2025-04-18","arxiv_id":"2504.13981","n_code_links":0,"syntology":null},{"paper":null,"title":"Learning from Streaming Video with Orthogonal Gradients","date":"2025-04-02","arxiv_id":"2504.01961","n_code_links":0,"syntology":null},{"paper":null,"title":"Chirp Localization via Fine-Tuned Transformer Model: A Proof-of-Concept Study","date":"2025-03-24","arxiv_id":"2503.22713","n_code_links":0,"syntology":null},{"paper":null,"title":"ARLED: Leveraging LED-based ARMAN Model for Abstractive Summarization of Persian Long Documents","date":"2025-03-13","arxiv_id":"2503.10233","n_code_links":0,"syntology":null},{"paper":null,"title":"Fine-Tuning Florence2 for Enhanced Object Detection in Un-constructed Environments: Vision-Language Model Approach","date":"2025-03-06","arxiv_id":"2503.04918","n_code_links":0,"syntology":null},{"paper":null,"title":"The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training","date":"2025-02-26","arxiv_id":"2502.19002","n_code_links":0,"syntology":null},{"paper":"/paper/cosmos-a-hybrid-adaptive-optimizer-for-memory","title":"COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMs","date":"2025-02-24","arxiv_id":"2502.17410","n_code_links":1,"syntology":null},{"paper":"/paper/muon-is-scalable-for-llm-training","title":"Muon is Scalable for LLM Training","date":"2025-02-24","arxiv_id":"2502.16982","n_code_links":1,"syntology":null},{"paper":null,"title":"Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs","date":"2025-02-21","arxiv_id":"2502.15938","n_code_links":0,"syntology":null},{"paper":"/paper/a-stronger-mixture-of-low-rank-experts-for","title":"A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models","date":"2025-02-20","arxiv_id":"2502.15828","n_code_links":1,"syntology":{"ran":3,"of":4,"unverified":1,"pointer_only":4}},{"paper":"/paper/accelerating-neural-network-training-an","title":"Accelerating Neural Network Training: An Analysis of the AlgoPerf Competition","date":"2025-02-20","arxiv_id":"2502.15015","n_code_links":2,"syntology":{"ran":0,"of":19,"unverified":19,"pointer_only":0}},{"paper":null,"title":"AdaGC: Improving Training Stability for Large Language Model Pretraining","date":"2025-02-16","arxiv_id":"2502.11034","n_code_links":0,"syntology":null},{"paper":null,"title":"FlexDeMo: Decoupled Momentum Optimization for Hybrid Sharded Data Parallel Training","date":"2025-02-10","arxiv_id":"2502.06728","n_code_links":0,"syntology":null},{"paper":null,"title":"Avoiding spurious sharpness minimization broadens applicability of SAM","date":"2025-02-04","arxiv_id":"2502.02407","n_code_links":0,"syntology":null},{"paper":null,"title":"How Memory in Optimization Algorithms Implicitly Modifies the Loss","date":"2025-02-04","arxiv_id":"2502.02132","n_code_links":0,"syntology":null},{"paper":null,"title":"The Ball-Proximal (=\"Broximal\") Point Method: a New Algorithm, Convergence Theory, and Applications","date":"2025-02-04","arxiv_id":"2502.02002","n_code_links":0,"syntology":null},{"paper":null,"title":"AlphaAdam:Asynchronous Masked Optimization with Dynamic Alpha for Selective Updates","date":"2025-01-30","arxiv_id":"2501.18094","n_code_links":0,"syntology":null},{"paper":"/paper/approximate-message-passing-for-bayesian","title":"Approximate Message Passing for Bayesian Neural Networks","date":"2025-01-26","arxiv_id":"2501.15573","n_code_links":1,"syntology":null},{"paper":null,"title":"Online Reinforcement Learning-Based Dynamic Adaptive Evaluation Function for Real-Time Strategy Tasks","date":"2025-01-07","arxiv_id":"2501.03824","n_code_links":0,"syntology":null},{"paper":"/paper/no-more-adam-learning-rate-scaling-at","title":"No More Adam: Learning Rate Scaling at Initialization is All You Need","date":"2024-12-16","arxiv_id":"2412.11768","n_code_links":1,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/language-modelling","name":"Language Modelling","papers":24},{"task":"/task/language-modeling","name":"Language Modeling","papers":22},{"task":"/task/image-classification","name":"Image Classification","papers":17},{"task":"/task/image-classification","name":"image-classification","papers":17},{"task":"/task/decoder","name":"Decoder","papers":12},{"task":"/task/question-answering","name":"Question Answering","papers":11},{"task":"/task/document-classification","name":"Document Classification","papers":10},{"task":null,"name":"GPU","papers":10},{"task":"/task/sentence","name":"Sentence","papers":10},{"task":"/task/object-detection","name":"Object Detection","papers":9},{"task":"/task/abstractive-text-summarization","name":"Abstractive Text Summarization","papers":8},{"task":"/task/classification-1","name":"Classification","papers":8},{"task":"/task/articles","name":"Articles","papers":7},{"task":"/task/deep-learning","name":"Deep Learning","papers":7},{"task":"/task/text-classification","name":"Text Classification","papers":7},{"task":"/task/natural-language-inference","name":"Natural Language Inference","papers":6},{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":6},{"task":"/task/object-detection-1","name":"object-detection","papers":6},{"task":"/task/segmentation","name":"Segmentation","papers":5},{"task":"/task/text-summarization","name":"Text Summarization","papers":5}],"tasks_shown":20,"n_tasks":207,"usage_by_year":[{"year":"2017","papers":1},{"year":"2018","papers":1},{"year":"2019","papers":2},{"year":"2020","papers":10},{"year":"2021","papers":24},{"year":"2022","papers":30},{"year":"2023","papers":62},{"year":"2024","papers":47},{"year":"2025","papers":29}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/adamw"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}