{"url":"/method/adam-mini","slug":"adam-mini","name":"Adam-mini","full_name":"Adaptive Moment Estimation - Mini","full_name_withheld":false,"description_markdown":"**Adam-mini** is a memory-efficient [Adam](https://paperswithcode.com/method/adam) variant that achieves on-par or better performance than [AdamW](https://paperswithcode.com/method/adamw) with 45% to 50% less memory footprint. Adam-mini reduces the memory footprint by cutting down the learning rate resources in Adam (i.e., $ {1}/{\\sqrt{v}} $). The authors find that ≥ 90% of these learning rates in $ v $ could be harmlessly removed if they (1) carefully partition the parameters into blocks following their proposed principle on Hessian structure; (2) assign a single but good learning rate to each parameter block. They further find that, for each of these parameter blocks, there exists a single high-quality learning rate that can outperform Adam, provided that sufficient resources are available to search it out.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Adam-mini: Use Fewer Learning Rates To Gain More","paper":"/paper/adam-mini-use-fewer-learning-rates-to-gain","first_author":"Yushun Zhang","n_authors":9,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/adam-mini-use-fewer-learning-rates-to-gain"},"source":{"url":"https://arxiv.org/abs/2406.16793v6","title":"Adam-mini: Use Fewer Learning Rates To Gain More","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/zyushun/Adam-mini/blob/main/adam_mini/adam_mini.py","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Optimization","url":"/methods/category/optimization","pwc_aliases":[]},{"area":"General","area_id":"general","collection":"Stochastic Optimization","url":"/methods/category/stochastic-optimization","pwc_aliases":[]}],"n_papers_tagged":2,"archive_num_papers":2,"papers_newest_first":[{"paper":null,"title":"The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training","date":"2025-02-26","arxiv_id":"2502.19002","n_code_links":0,"syntology":null},{"paper":"/paper/adam-mini-use-fewer-learning-rates-to-gain","title":"Adam-mini: Use Fewer Learning Rates To Gain More","date":"2024-06-24","arxiv_id":"2406.16793","n_code_links":1,"syntology":{"ran":4,"of":4,"unverified":0,"pointer_only":4}}],"papers_shown":2,"tasks":[{"task":"/task/language-modeling","name":"Language Modeling","papers":1},{"task":"/task/language-modelling","name":"Language Modelling","papers":1},{"task":"/task/large-language-model","name":"Large Language Model","papers":1}],"tasks_shown":3,"n_tasks":3,"usage_by_year":[{"year":"2024","papers":1},{"year":"2025","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/adam-mini"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}