{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-marginal-value-of-adaptive-gradient","title":"The Marginal Value of Adaptive Gradient Methods in Machine Learning","arxiv_id":"1705.08292","date":"2017-05-23","proceeding":"NeurIPS 2017 12","authors":["Ashia C. Wilson","Rebecca Roelofs","Mitchell Stern","Nathan Srebro","Benjamin Recht"],"abstract":"Adaptive optimization methods, which perform local optimization with a metric\nconstructed from the history of iterates, are becoming increasingly popular for\ntraining deep neural networks. Examples include AdaGrad, RMSProp, and Adam. We\nshow that for simple overparameterized problems, adaptive methods often find\ndrastically different solutions than gradient descent (GD) or stochastic\ngradient descent (SGD). We construct an illustrative binary classification\nproblem where the data is linearly separable, GD and SGD achieve zero test\nerror, and AdaGrad, Adam, and RMSProp attain test errors arbitrarily close to\nhalf. We additionally study the empirical generalization capability of adaptive\nmethods on several state-of-the-art deep learning models. We observe that the\nsolutions found by adaptive methods generalize worse (often significantly\nworse) than SGD, even when these solutions have better training performance.\nThese results suggest that practitioners should reconsider the use of adaptive\nmethods to train neural networks.","url_abs":"http://arxiv.org/abs/1705.08292v2","url_pdf":"http://arxiv.org/pdf/1705.08292v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-marginal-value-of-adaptive-gradient","repo_url":"https://github.com/jrzech/reproduce-chexnet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"the-marginal-value-of-adaptive-gradient","repo_url":"https://github.com/pasta41/deception","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"the-marginal-value-of-adaptive-gradient","repo_url":"https://github.com/vrunm/text-classification-financial-phrase-bank","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"},{"task_slug":"binary-classification","task_name":"Binary Classification"}],"methods":[{"method_slug":"adagrad","method_name":"AdaGrad"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"rmsprop","method_name":"RMSProp"},{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1705.08292","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}