Methods › General › Stochastic Optimization › AdamW
AdamW
Introduced by Ilya Loshchilov et al. in Decoupled Weight Decay Regularization
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
AdamW is a stochastic optimization method that modifies the typical implementation of weight decay in Adam, by decoupling weight decay from the gradient update. To see this, L₂ regularization in Adam is usually implemented with the below modification where wₜ is the rate of the weight decay at time t:
gₜ = ∇f(θₜ) + wₜθₜ
while AdamW adjusts the weight decay term to appear in the gradient update:
θ_(t+1, i) = θ_(t, i) - η(1/(√(v̂ₜ + ϵ))·m̂ₜ + w_(t, i)θ_(t, i)), ∀t
In Syntology name not resolved (unknown_name). The lookup matched no method, paper or venue by this name; that asserts nothing about whether code exists.
Near matches the lookup lists, each a different name: AdamW-GS; DP-AdamW-BC; QEF-GT-AdamW; Stiefel-AdamW; AdamW-influence; AdamW-Adafactor hybrid; Shape-Boundary Orthogonal AdamW (SBO-AdamW).
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 206 · with a code link: 114 · where Syntology ran a sample: 43 (37 with a run with no instrument failure, 6 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (43 of 206 tagged: 37 with a run with no instrument failure, 6 where every run was a failure of Syntology's instrument)
Page 1 of 3: papers 1 to 100 of 206, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
I Know Which LLM Wrote Your Code Last Summer: LLM generated Code Stylometry for Authorship Attribution 18 Jun 2025 · 0 repositories · arXiv:2506.17323
-
Improving LoRA with Variational Learning 17 Jun 2025 · 0 repositories · arXiv:2506.14280
-
PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective 27 May 2025 · 0 repositories · arXiv:2505.21799
-
AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training 22 May 2025 · 1 repository · arXiv:2505.16363Syntology official: no sample here; runs from other or unrecorded repositories · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
Enhancing Abstractive Summarization of Scientific Papers Using Structure Information 20 May 2025 · 1 repository · arXiv:2505.14179
-
A Physics-Inspired Optimizer: Velocity Regularized Adam 19 May 2025 · 0 repositories · arXiv:2505.13196
-
Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training 19 May 2025 · 0 repositories · arXiv:2505.13738
-
On the O((√(d))/(K^(1/4))) Convergence Rate of AdamW Measured by ℓ₁ Norm 17 May 2025 · 0 repositories · arXiv:2505.11840
-
Variational Visual Question Answering 14 May 2025 · 0 repositories · arXiv:2505.09591
-
Practical Efficiency of Muon for Pretraining 4 May 2025 · 1 repository · arXiv:2505.02222
-
CacheFormer: High Attention-Based Segment Caching 18 Apr 2025 · 0 repositories · arXiv:2504.13981
-
Learning from Streaming Video with Orthogonal Gradients 2 Apr 2025 · 0 repositories · arXiv:2504.01961
-
Chirp Localization via Fine-Tuned Transformer Model: A Proof-of-Concept Study 24 Mar 2025 · 0 repositories · arXiv:2503.22713
-
ARLED: Leveraging LED-based ARMAN Model for Abstractive Summarization of Persian Long Documents 13 Mar 2025 · 0 repositories · arXiv:2503.10233
-
Fine-Tuning Florence2 for Enhanced Object Detection in Un-constructed Environments: Vision-Language Model Approach 6 Mar 2025 · 0 repositories · arXiv:2503.04918
-
The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training 26 Feb 2025 · 0 repositories · arXiv:2502.19002
-
COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMs 24 Feb 2025 · 1 repository · arXiv:2502.17410
-
Muon is Scalable for LLM Training 24 Feb 2025 · 1 repository · arXiv:2502.16982
-
Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs 21 Feb 2025 · 0 repositories · arXiv:2502.15938
-
A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models 20 Feb 2025 · 1 repository · arXiv:2502.15828Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified (of 4 harvested samples) · 4 pointer-only (licence)
-
Accelerating Neural Network Training: An Analysis of the AlgoPerf Competition 20 Feb 2025 · 2 repositories · arXiv:2502.15015Syntology 16 ran (of which 0 constructed an object rather than computing a result; 16 with no instrument failure: 0 honoured, 0 violated, 16 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified (of 19 harvested samples)
-
AdaGC: Improving Training Stability for Large Language Model Pretraining 16 Feb 2025 · 0 repositories · arXiv:2502.11034
-
FlexDeMo: Decoupled Momentum Optimization for Hybrid Sharded Data Parallel Training 10 Feb 2025 · 0 repositories · arXiv:2502.06728
-
Avoiding spurious sharpness minimization broadens applicability of SAM 4 Feb 2025 · 0 repositories · arXiv:2502.02407
-
How Memory in Optimization Algorithms Implicitly Modifies the Loss 4 Feb 2025 · 0 repositories · arXiv:2502.02132
-
The Ball-Proximal (="Broximal") Point Method: a New Algorithm, Convergence Theory, and Applications 4 Feb 2025 · 0 repositories · arXiv:2502.02002
-
AlphaAdam:Asynchronous Masked Optimization with Dynamic Alpha for Selective Updates 30 Jan 2025 · 0 repositories · arXiv:2501.18094
-
Approximate Message Passing for Bayesian Neural Networks 26 Jan 2025 · 1 repository · arXiv:2501.15573
-
Online Reinforcement Learning-Based Dynamic Adaptive Evaluation Function for Real-Time Strategy Tasks 7 Jan 2025 · 0 repositories · arXiv:2501.03824
-
No More Adam: Learning Rate Scaling at Initialization is All You Need 16 Dec 2024 · 1 repository · arXiv:2412.11768
-
APOLLO: SGD-like Memory, AdamW-level Performance 6 Dec 2024 · 1 repository · arXiv:2412.05270Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 2 honoured, 0 violated, 2 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified (of 9 harvested samples) · 9 pointer-only (licence)
-
DeMo: Decoupled Momentum Optimization 29 Nov 2024 · 1 repository · arXiv:2411.19870
-
COAP: Memory-Efficient Training with Correlation-Aware Gradient Projection 26 Nov 2024 · 0 repositories · arXiv:2412.00071
-
Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise 24 Nov 2024 · 0 repositories · arXiv:2411.15958
-
MARS: Unleashing the Power of Variance Reduction for Training Large Models 15 Nov 2024 · 2 repositories · arXiv:2411.10438Syntology official (archive's flag): 2 ran · 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified (of 5 harvested samples) · 1 pointer-only (licence)
-
Continuous-Time Analysis of Adaptive Optimization and Normalization 8 Nov 2024 · 0 repositories · arXiv:2411.05746
-
Variational Low-Rank Adaptation Using IVON 7 Nov 2024 · 1 repository · arXiv:2411.04421Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (of 4 harvested samples) · 4 pointer-only (licence)
-
Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models 3 Nov 2024 · 2 repositories · arXiv:2411.01713Syntology official (archive's flag): 1 ran · 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result (of 1 harvested sample) · 1 pointer-only (licence)
-
Understanding Players as if They Are Talking to the Game in a Customized Language: A Pilot Study 24 Oct 2024 · 0 repositories · arXiv:2410.18605
-
Natural GaLore: Accelerating GaLore for memory-efficient LLM Training and Fine-tuning 21 Oct 2024 · 1 repository · arXiv:2410.16029
-
Extra Global Attention Designation Using Keyword Detection in Sparse Transformer Architectures 11 Oct 2024 · 0 repositories · arXiv:2410.08971
-
Learning Low-Level Causal Relations using a Simulated Robotic Arm 10 Oct 2024 · 1 repository · arXiv:2410.07751
-
The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Models 9 Oct 2024 · 1 repository · arXiv:2410.06554Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 5 where Syntology's instrument failed) · 4 unverified (of 9 harvested samples)
-
A second-order-like optimizer with adaptive gradient scaling for deep learning 8 Oct 2024 · 1 repository · arXiv:2410.05871
-
How Much Can We Forget about Data Contamination? 4 Oct 2024 · 1 repository · arXiv:2410.03249Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 2 harvested samples)
-
SOAP: Improving and Stabilizing Shampoo using Adam 17 Sep 2024 · 1 repository · arXiv:2409.11321
-
Self-Supervised Syllable Discovery Based on Speaker-Disentangled HuBERT 16 Sep 2024 · 1 repository · arXiv:2409.10103
-
The AdEMAMix Optimizer: Better, Faster, Older 5 Sep 2024 · 4 repositories · arXiv:2409.03137Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 2 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 4 harvested samples) · 4 pointer-only (licence)
-
Beat this! Accurate beat tracking without DBN postprocessing 31 Jul 2024 · 1 repository · arXiv:2407.21658
-
Adaptive Gradient Regularization: A Faster and Generalizable Optimization Technique for Deep Neural Networks 24 Jul 2024 · 0 repositories · arXiv:2407.16944
-
Resolving Discrepancies in Compute-Optimal Scaling of Language Models 27 Jun 2024 · 3 repositories · arXiv:2406.19146Syntology official (archive's flag): 9 ran · 10 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 3 violated, 0 with no contract checked; 5 where Syntology's instrument failed) · 6 unverified (of 16 harvested samples) · 4 pointer-only (licence)
-
Adam-mini: Use Fewer Learning Rates To Gain More 24 Jun 2024 · 1 repository · arXiv:2406.16793Syntology official (archive's flag): 3 ran · 4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 4 harvested samples) · 4 pointer-only (licence)
-
Separate and Reconstruct: Asymmetric Encoder-Decoder for Speech Separation 10 Jun 2024 · 1 repository · arXiv:2406.05983
-
RICo: Reddit ideological communities 5 Jun 2024 · 1 repository
-
Improving Generalization and Convergence by Enhancing Implicit Regularization 31 May 2024 · 1 repository · arXiv:2405.20763Syntology official (archive's flag): 7 ran · 7 ran (of which 1 constructed an object rather than computing a result; 4 with no instrument failure: 2 honoured, 0 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 7 harvested samples) · 7 pointer-only (licence)
-
The Road Less Scheduled 24 May 2024 · 2 repositories · arXiv:2405.15682
-
How to set AdamW's weight decay as you scale model and dataset size 22 May 2024 · 0 repositories · arXiv:2405.13698
-
Robust Hyperbolic Learning with Curvature-Aware Optimization 22 May 2024 · 0 repositories · arXiv:2405.13979
-
Transfer Learning in Pre-Trained Large Language Models for Malware Detection Based on System Calls 15 May 2024 · 0 repositories · arXiv:2405.09318
-
Audio-Visual Speech Recognition based on Regulated Transformer and Spatio-Temporal Fusion Strategy for Driver Assistive Systems 9 May 2024 · 1 repository
-
FashionFail: Addressing Failure Cases in Fashion Object Detection and Segmentation 12 Apr 2024 · 1 repository · arXiv:2404.08582
-
Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimization 5 Apr 2024 · 0 repositories · arXiv:2404.04454
-
Advancing AI with Integrity: Ethical Challenges and Solutions in Neural Machine Translation 1 Apr 2024 · 0 repositories · arXiv:2404.01070
-
Communication Efficient Distributed Training with Distributed Lion 30 Mar 2024 · 0 repositories · arXiv:2404.00438
-
LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning 26 Mar 2024 · 1 repository · arXiv:2403.17919Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample)
-
Adapprox: Adaptive Approximation in Adam Optimization via Randomized Low-Rank Matrices 22 Mar 2024 · 0 repositories · arXiv:2403.14958
-
AdaFish: Fast low-rank parameter-efficient fine-tuning by using second-order information 19 Mar 2024 · 0 repositories · arXiv:2403.13128
-
A multi-cohort study on prediction of acute brain dysfunction states using selective state space models 11 Mar 2024 · 0 repositories · arXiv:2403.07201
-
Inverse-Free Fast Natural Gradient Descent Method for Deep Learning 6 Mar 2024 · 0 repositories · arXiv:2403.03473
-
Adaptation of Biomedical and Clinical Pretrained Models to French Long Documents: A Comparative Study 26 Feb 2024 · 1 repository · arXiv:2402.16689
-
Should I try multiple optimizers when fine-tuning pre-trained Transformers for NLP tasks? Should I tune their hyperparameters? 10 Feb 2024 · 0 repositories · arXiv:2402.06948
-
Accurate and Well-Calibrated ICD Code Assignment Through Attention Over Diverse Label Embeddings 5 Feb 2024 · 1 repository · arXiv:2402.03172Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 4 where Syntology's instrument failed) · 0 unverified (of 7 harvested samples)
-
UniMem: Towards a Unified View of Long-Context Large Language Models 5 Feb 2024 · 1 repository · arXiv:2402.03009
-
Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models 4 Feb 2024 · 1 repository · arXiv:2402.02347Syntology official (archive's flag): 12 ran · 15 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 2 honoured, 1 violated, 6 with no contract checked; 6 where Syntology's instrument failed) · 6 unverified (of 21 harvested samples) · 3 pointer-only (licence)
-
HiFT: A Hierarchical Full Parameter Fine-Tuning Strategy 26 Jan 2024 · 1 repository · arXiv:2401.15207Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples)
-
Enhanced Labeling Technique for Reddit Text and Fine-Tuned Longformer Models for Classifying Depression Severity in English and Luganda 25 Jan 2024 · 0 repositories · arXiv:2401.14240
-
Exploring Automatic Text Simplification of German Narrative Documents 15 Dec 2023 · 1 repository · arXiv:2312.09907
-
Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFAC 9 Dec 2023 · 2 repositories · arXiv:2312.05705Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
LLVMs4Protest: Harnessing the Power of Large Language and Vision Models for Deciphering Protests in the News 30 Nov 2023 · 1 repository · arXiv:2311.18241
-
Weight Norm Control 19 Nov 2023 · 0 repositories · arXiv:2311.11446
-
OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoning 16 Nov 2023 · 1 repository · arXiv:2311.09724Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
DiLoCo: Distributed Low-Communication Training of Language Models 14 Nov 2023 · 0 repositories · arXiv:2311.08105
-
SynthEnsemble: A Fusion of CNN, Vision Transformer, and Hybrid Models for Multi-Label Chest X-Ray Classification 13 Nov 2023 · 1 repository · arXiv:2311.07750
-
A Quadratic Synchronization Rule for Distributed Deep Learning 22 Oct 2023 · 1 repository · arXiv:2310.14423Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 6 harvested samples)
-
Towards Harmful Erotic Content Detection through Coreference-Driven Contextual Analysis 22 Oct 2023 · 0 repositories · arXiv:2310.14325
-
FerNeXt: Facial Expression Recognition Using ConvNeXt with Channel Attention 20 Oct 2023 · 1 repository
-
Multi-level Contrastive Learning for Script-based Character Understanding 20 Oct 2023 · 1 repository · arXiv:2310.13231
-
Improving Long Document Topic Segmentation Models With Enhanced Coherence Modeling 18 Oct 2023 · 1 repository · arXiv:2310.11772
-
Jorge: Approximate Preconditioning for GPU-efficient Second-order Optimization 18 Oct 2023 · 0 repositories · arXiv:2310.12298
-
AdaLomo: Low-memory Optimization with Adaptive Learning Rate 16 Oct 2023 · 1 repository · arXiv:2310.10195Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)
-
Adam-family Methods with Decoupled Weight Decay in Deep Learning 13 Oct 2023 · 0 repositories · arXiv:2310.08858
-
Lion Secretly Solves Constrained Optimization: As Lyapunov Predicts 9 Oct 2023 · 0 repositories · arXiv:2310.05898
-
Combining UPerNet and ConvNeXt for Contrails Identification to reduce Global Warming 7 Oct 2023 · 0 repositories · arXiv:2310.04808
-
Hallucination Reduction in Long Input Text Summarization 28 Sep 2023 · 1 repository · arXiv:2309.16781
-
An NLP Benchmark Dataset for Assessing Corporate Climate Policy Engagement 26 Sep 2023 · 0 repositories
-
Graph-enhanced Optimizers for Structure-aware Recommendation Embedding Evolution 24 Sep 2023 · 1 repository · arXiv:2310.03032Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)
-
AdaPlus: Integrating Nesterov Momentum and Precise Stepsize Adjustment on AdamW Basis 5 Sep 2023 · 1 repository · arXiv:2309.01966
-
Language Models for Novelty Detection in System Call Traces 5 Sep 2023 · 0 repositories · arXiv:2309.02206
-
Local Large Language Models for Complex Structured Medical Tasks 3 Aug 2023 · 1 repository · arXiv:2308.01727
-
Can Model Fusing Help Transformers in Long Document Classification? An Empirical Study 18 Jul 2023 · 1 repository · arXiv:2307.09532
Tasks archive 2025-07-28
20 shown of 207 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections