Methods › General › Stochastic Optimization › AdamW
AdamW
Introduced by Ilya Loshchilov et al. in Decoupled Weight Decay Regularization
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
AdamW is a stochastic optimization method that modifies the typical implementation of weight decay in Adam, by decoupling weight decay from the gradient update. To see this, L₂ regularization in Adam is usually implemented with the below modification where wₜ is the rate of the weight decay at time t:
gₜ = ∇f(θₜ) + wₜθₜ
while AdamW adjusts the weight decay term to appear in the gradient update:
θ_(t+1, i) = θ_(t, i) - η(1/(√(v̂ₜ + ϵ))·m̂ₜ + w_(t, i)θ_(t, i)), ∀t
Papers archive 2025-07-28
30 shown of 206, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
I Know Which LLM Wrote Your Code Last Summer: LLM generated Code Stylometry for Authorship Attribution 18 Jun 2025 · 0 repositories · arXiv:2506.17323
-
Improving LoRA with Variational Learning 17 Jun 2025 · 0 repositories · arXiv:2506.14280
-
PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective 27 May 2025 · 0 repositories · arXiv:2505.21799
-
AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training 22 May 2025 · 1 repository · arXiv:2505.16363Syntology ran 0 of 1 samples · 1 unverified
-
Enhancing Abstractive Summarization of Scientific Papers Using Structure Information 20 May 2025 · 1 repository · arXiv:2505.14179
-
A Physics-Inspired Optimizer: Velocity Regularized Adam 19 May 2025 · 0 repositories · arXiv:2505.13196
-
Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training 19 May 2025 · 0 repositories · arXiv:2505.13738
-
On the O((√(d))/(K^(1/4))) Convergence Rate of AdamW Measured by ℓ₁ Norm 17 May 2025 · 0 repositories · arXiv:2505.11840
-
Variational Visual Question Answering 14 May 2025 · 0 repositories · arXiv:2505.09591
-
Practical Efficiency of Muon for Pretraining 4 May 2025 · 1 repository · arXiv:2505.02222
-
CacheFormer: High Attention-Based Segment Caching 18 Apr 2025 · 0 repositories · arXiv:2504.13981
-
Learning from Streaming Video with Orthogonal Gradients 2 Apr 2025 · 0 repositories · arXiv:2504.01961
-
Chirp Localization via Fine-Tuned Transformer Model: A Proof-of-Concept Study 24 Mar 2025 · 0 repositories · arXiv:2503.22713
-
ARLED: Leveraging LED-based ARMAN Model for Abstractive Summarization of Persian Long Documents 13 Mar 2025 · 0 repositories · arXiv:2503.10233
-
Fine-Tuning Florence2 for Enhanced Object Detection in Un-constructed Environments: Vision-Language Model Approach 6 Mar 2025 · 0 repositories · arXiv:2503.04918
-
The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training 26 Feb 2025 · 0 repositories · arXiv:2502.19002
-
COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMs 24 Feb 2025 · 1 repository · arXiv:2502.17410
-
Muon is Scalable for LLM Training 24 Feb 2025 · 1 repository · arXiv:2502.16982
-
Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs 21 Feb 2025 · 0 repositories · arXiv:2502.15938
-
A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models 20 Feb 2025 · 1 repository · arXiv:2502.15828Syntology ran 3 of 4 samples · 1 unverified · 4 pointer-only (licence)
-
Accelerating Neural Network Training: An Analysis of the AlgoPerf Competition 20 Feb 2025 · 2 repositories · arXiv:2502.15015Syntology ran 0 of 19 samples · 19 unverified
-
AdaGC: Improving Training Stability for Large Language Model Pretraining 16 Feb 2025 · 0 repositories · arXiv:2502.11034
-
FlexDeMo: Decoupled Momentum Optimization for Hybrid Sharded Data Parallel Training 10 Feb 2025 · 0 repositories · arXiv:2502.06728
-
Avoiding spurious sharpness minimization broadens applicability of SAM 4 Feb 2025 · 0 repositories · arXiv:2502.02407
-
How Memory in Optimization Algorithms Implicitly Modifies the Loss 4 Feb 2025 · 0 repositories · arXiv:2502.02132
-
The Ball-Proximal (="Broximal") Point Method: a New Algorithm, Convergence Theory, and Applications 4 Feb 2025 · 0 repositories · arXiv:2502.02002
-
AlphaAdam:Asynchronous Masked Optimization with Dynamic Alpha for Selective Updates 30 Jan 2025 · 0 repositories · arXiv:2501.18094
-
Approximate Message Passing for Bayesian Neural Networks 26 Jan 2025 · 1 repository · arXiv:2501.15573
-
Online Reinforcement Learning-Based Dynamic Adaptive Evaluation Function for Real-Time Strategy Tasks 7 Jan 2025 · 0 repositories · arXiv:2501.03824
-
No More Adam: Learning Rate Scaling at Initialization is All You Need 16 Dec 2024 · 1 repository · arXiv:2412.11768
Tasks archive 2025-07-28
20 shown of 207 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections