{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-the-convergence-of-adam-and-beyond-1","title":"On the Convergence of Adam and Beyond","arxiv_id":"1904.09237","date":"2019-04-19","proceeding":"ICLR 2018 1","authors":["Sashank J. Reddi","Satyen Kale","Sanjiv Kumar"],"abstract":"Several recently proposed stochastic optimization methods that have been\nsuccessfully used in training deep networks such as RMSProp, Adam, Adadelta,\nNadam are based on using gradient updates scaled by square roots of exponential\nmoving averages of squared past gradients. In many applications, e.g. learning\nwith large output spaces, it has been empirically observed that these\nalgorithms fail to converge to an optimal solution (or a critical point in\nnonconvex settings). We show that one cause for such failures is the\nexponential moving average used in the algorithms. We provide an explicit\nexample of a simple convex optimization setting where Adam does not converge to\nthe optimal solution, and describe the precise problems with the previous\nanalysis of Adam algorithm. Our analysis suggests that the convergence issues\ncan be fixed by endowing such algorithms with `long-term memory' of past\ngradients, and propose new variants of the Adam algorithm which not only fix\nthe convergence issues but often also lead to improved empirical performance.","url_abs":"http://arxiv.org/abs/1904.09237v1","url_pdf":"http://arxiv.org/pdf/1904.09237v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-the-convergence-of-adam-and-beyond-1","repo_url":"https://github.com/brett-daley/expectigrad","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"on-the-convergence-of-adam-and-beyond-1","repo_url":"https://github.com/MS-Mind/MS-Code-05","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":{"status":"ok"}},{"paper_slug":"on-the-convergence-of-adam-and-beyond-1","repo_url":"https://github.com/labmlai/annotated_deep_learning_paper_implementations","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"stochastic-optimization","task_name":"Stochastic Optimization"}],"methods":[{"method_slug":"amsgrad","method_name":"AMSGrad"},{"method_slug":"adadelta","method_name":"AdaDelta"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"nadam","method_name":"NADAM"},{"method_slug":"rmsprop","method_name":"RMSProp"}],"datasets_introduced":[],"methods_introduced":[{"slug":"amsgrad","name":"AMSGrad","full_name":"AMSGrad"}],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1904.09237","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1904.09237"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MS-Mind/MS-Code-05","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/brett-daley/expectigrad","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/labmlai/annotated_deep_learning_paper_implementations","reach":null}],"summary":{"unverified":2},"by_repo_kind":{"listed":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"54eb7c2a26ec096d","entry":"generate_test_sequence","repo":"brett-daley/expectigrad","repo_kind":"listed","path":"expectigrad/testing.py","file_url":"https://github.com/brett-daley/expectigrad/blob/HEAD/expectigrad/testing.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"54eb7c2a26ec096d"}},{"code_sha256_prefix":"05b3b6f5fa850f2f","entry":"gradient_test_function","repo":"brett-daley/expectigrad","repo_kind":"listed","path":"expectigrad/testing.py","file_url":"https://github.com/brett-daley/expectigrad/blob/HEAD/expectigrad/testing.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"05b3b6f5fa850f2f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}