{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/adam-mini-use-fewer-learning-rates-to-gain","title":"Adam-mini: Use Fewer Learning Rates To Gain More","arxiv_id":"2406.16793","date":"2024-06-24","proceeding":null,"authors":["Yushun Zhang","Congliang Chen","Ziniu Li","Tian Ding","Chenwei Wu","Diederik P. Kingma","Yinyu Ye","Zhi-Quan Luo","Ruoyu Sun"],"abstract":"We propose Adam-mini, an optimizer that achieves on par or better performance than AdamW with 50% less memory footprint. Adam-mini reduces memory by cutting down the learning rate resources in Adam (i.e., $1/\\sqrt{v}$). By investigating the Hessian structure of neural nets, we find Adam's $v$ might not function at its full potential as effectively as we expected. We find that $\\geq$ 99.9% of these learning rates in $v$ could be harmlessly removed if we (1) carefully partition the parameters into blocks following our new principle on Hessian structure; (2) assign a single but good learning rate to each parameter block. We then provide one simple way to find good learning rates and propose Adam-mini. Empirically, we verify that Adam-mini performs on par or better than AdamW on various language models sized from 39M to 13B for pre-training, supervised fine-tuning, and RLHF. The reduced memory footprint of Adam-mini also alleviates communication overheads among GPUs, thereby increasing throughput. For instance, Adam-mini achieves 49.6% higher throughput than AdamW when pre-training Llama 2-7B on $2\\times$ A800-80GB GPUs, which saves 33% wall-clock time for pre-training.","url_abs":"https://arxiv.org/abs/2406.16793v6","url_pdf":"https://arxiv.org/pdf/2406.16793v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"adam-mini-use-fewer-learning-rates-to-gain","repo_url":"https://github.com/zyushun/adam-mini","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"adam-mini","method_name":"Adam-mini"},{"method_slug":"adamw","method_name":"AdamW"},{"method_slug":"llama","method_name":"LLaMA"}],"datasets_introduced":[],"methods_introduced":[{"slug":"adam-mini","name":"Adam-mini","full_name":"Adaptive Moment Estimation - Mini"}],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2406.16793","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.16793"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zyushun/adam-mini","reach":null}],"summary":{"ran_fixture":3,"ran_violates":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"f5349a28fe475e9b","entry":"apply_rotary_emb","repo":"zyushun/adam-mini","repo_kind":"official","path":"examples/llama/torchtitan/models/llama/model.py","file_url":"https://github.com/zyushun/adam-mini/blob/HEAD/examples/llama/torchtitan/models/llama/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f5349a28fe475e9b"}},{"code_sha256_prefix":"2f0a690c70d11c23","entry":"gather_log_probs","repo":"zyushun/adam-mini","repo_kind":"official","path":"examples/RLHF/dschat/remax/remax_trainer.py","file_url":"https://github.com/zyushun/adam-mini/blob/HEAD/examples/RLHF/dschat/remax/remax_trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2f0a690c70d11c23"}},{"code_sha256_prefix":"1a557a96fabbca72","entry":"precompute_freqs_cis","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"1a557a96fabbca72"}},{"code_sha256_prefix":"c7b1e7c860ab5a79","entry":"reshape_for_broadcast","repo":"zyushun/adam-mini","repo_kind":"official","path":"examples/llama/torchtitan/models/llama/model.py","file_url":"https://github.com/zyushun/adam-mini/blob/HEAD/examples/llama/torchtitan/models/llama/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c7b1e7c860ab5a79"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}