{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/when-can-you-get-away-with-low-memory-adam","title":"When Can You Get Away with Low Memory Adam?","arxiv_id":"2503.01843","date":"2025-03-03","proceeding":null,"authors":["Dayal Singh Kalra","John Kirchenbauer","Maissam Barkeshli","Tom Goldstein"],"abstract":"Adam is the go-to optimizer for training modern machine learning models, but it requires additional memory to maintain the moving averages of the gradients and their squares. While various low-memory optimizers have been proposed that sometimes match the performance of Adam, their lack of reliability has left Adam as the default choice. In this work, we apply a simple layer-wise Signal-to-Noise Ratio (SNR) analysis to quantify when second-moment tensors can be effectively replaced by their means across different dimensions. Our SNR analysis reveals how architecture, training hyperparameters, and dataset properties impact compressibility along Adam's trajectory, naturally leading to $\\textit{SlimAdam}$, a memory-efficient Adam variant. $\\textit{SlimAdam}$ compresses the second moments along dimensions with high SNR when feasible, and leaves when compression would be detrimental. Through experiments across a diverse set of architectures and training scenarios, we show that $\\textit{SlimAdam}$ matches Adam's performance and stability while saving up to $98\\%$ of total second moments. Code for $\\textit{SlimAdam}$ is available at https://github.com/dayal-kalra/low-memory-adam.","url_abs":"https://arxiv.org/abs/2503.01843v3","url_pdf":"https://arxiv.org/pdf/2503.01843v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"when-can-you-get-away-with-low-memory-adam","repo_url":"https://github.com/dayal-kalra/low-memory-adam","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2503.01843","atlas_url":"https://app.syntology.ai/?focus=2503.01843","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.01843"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dayal-kalra/low-memory-adam","reach":null}],"summary":{"ran_honours":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"507e1ecb6635c782","entry":"get_rule_for_parameter","repo":"dayal-kalra/low-memory-adam","repo_kind":"official","path":"utils/slimadam.py","file_url":"https://github.com/dayal-kalra/low-memory-adam/blob/HEAD/utils/slimadam.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"507e1ecb6635c782"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}