Methods › General › Optimization › Adam-mini
Adaptive Moment Estimation - Mini
Adam-mini
Introduced by Yushun Zhang et al. in Adam-mini: Use Fewer Learning Rates To Gain More
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Adam-mini is a memory-efficient Adam variant that achieves on-par or better performance than AdamW with 45% to 50% less memory footprint. Adam-mini reduces the memory footprint by cutting down the learning rate resources in Adam (i.e., 1/√(v)). The authors find that ≥ 90% of these learning rates in $ v $ could be harmlessly removed if they (1) carefully partition the parameters into blocks following their proposed principle on Hessian structure; (2) assign a single but good learning rate to each parameter block. They further find that, for each of these parameter blocks, there exists a single high-quality learning rate that can outperform Adam, provided that sufficient resources are available to search it out.
Papers archive 2025-07-28
2 shown of 2, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training 26 Feb 2025 · 0 repositories · arXiv:2502.19002
-
Adam-mini: Use Fewer Learning Rates To Gain More 24 Jun 2024 · 1 repository · arXiv:2406.16793Syntology ran 4 of 4 samples · 0 unverified · 4 pointer-only (licence)
Tasks archive 2025-07-28
3 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
| Task | Papers |
|---|---|
| Language Modeling | 1 |
| Language Modelling | 1 |
| Large Language Model | 1 |
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections