{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improving-generalization-performance-by","title":"Improving Generalization Performance by Switching from Adam to SGD","arxiv_id":"1712.07628","date":"2017-12-20","proceeding":null,"authors":["Nitish Shirish Keskar","Richard Socher"],"abstract":"Despite superior training outcomes, adaptive optimization methods such as\nAdam, Adagrad or RMSprop have been found to generalize poorly compared to\nStochastic gradient descent (SGD). These methods tend to perform well in the\ninitial portion of training but are outperformed by SGD at later stages of\ntraining. We investigate a hybrid strategy that begins training with an\nadaptive method and switches to SGD when appropriate. Concretely, we propose\nSWATS, a simple strategy which switches from Adam to SGD when a triggering\ncondition is satisfied. The condition we propose relates to the projection of\nAdam steps on the gradient subspace. By design, the monitoring process for this\ncondition adds very little overhead and does not increase the number of\nhyperparameters in the optimizer. We report experiments on several standard\nbenchmarks such as: ResNet, SENet, DenseNet and PyramidNet for the CIFAR-10 and\nCIFAR-100 data sets, ResNet on the tiny-ImageNet data set and language modeling\nwith recurrent networks on the PTB and WT2 data sets. The results show that our\nstrategy is capable of closing the generalization gap between SGD and Adam on a\nmajority of the tasks.","url_abs":"http://arxiv.org/abs/1712.07628v1","url_pdf":"http://arxiv.org/pdf/1712.07628v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improving-generalization-performance-by","repo_url":"https://github.com/BZbyr/SimpleRNN","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"improving-generalization-performance-by","repo_url":"https://github.com/dfan/single-image-surface-normal-estimation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"improving-generalization-performance-by","repo_url":"https://github.com/harruff/Senior_Project_Repository","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"improving-generalization-performance-by","repo_url":"https://github.com/hphuongdhsp/Kaggle-TGS-Salt-Identification-Challenge","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"improving-generalization-performance-by","repo_url":"https://github.com/mhmdsabry/Sentiment_Analysis","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"unanswered"}},{"paper_slug":"improving-generalization-performance-by","repo_url":"https://github.com/tr7200/Patent-Text-Analytics-and-Venture-backed-Innovation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"adagrad","method_name":"AdaGrad"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"bottleneck-residual-block","method_name":"Bottleneck Residual Block"},{"method_slug":"concatenated-skip-connection","method_name":"Concatenated Skip Connection"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-block","method_name":"Dense Block"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"kaiming-initialization","method_name":"Kaiming Initialization"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"pyramidnet","method_name":"PyramidNet"},{"method_slug":"pyramidal-bottleneck-residual-unit","method_name":"Pyramidal Bottleneck Residual Unit"},{"method_slug":"pyramidal-residual-unit","method_name":"Pyramidal Residual Unit"},{"method_slug":"rmsprop","method_name":"RMSProp"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-block","method_name":"Residual Block"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"senet","method_name":"SENet"},{"method_slug":"sgd","method_name":"SGD"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"squeeze-and-excitation-block","method_name":"Squeeze-and-Excitation Block"},{"method_slug":"zero-padded-shortcut-connection","method_name":"Zero-padded Shortcut Connection"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1712.07628","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}