{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/super-convergence-very-fast-training-of-1","title":"Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates","arxiv_id":"1708.07120","date":"2017-08-23","proceeding":null,"authors":["Leslie N. Smith","Nicholay Topin"],"abstract":"In this paper, we describe a phenomenon, which we named \"super-convergence\",\nwhere neural networks can be trained an order of magnitude faster than with\nstandard training methods. The existence of super-convergence is relevant to\nunderstanding why deep networks generalize well. One of the key elements of\nsuper-convergence is training with one learning rate cycle and a large maximum\nlearning rate. A primary insight that allows super-convergence training is that\nlarge learning rates regularize the training, hence requiring a reduction of\nall other forms of regularization in order to preserve an optimal\nregularization balance. We also derive a simplification of the Hessian Free\noptimization method to compute an estimate of the optimal learning rate.\nExperiments demonstrate super-convergence for Cifar-10/100, MNIST and Imagenet\ndatasets, and resnet, wide-resnet, densenet, and inception architectures. In\naddition, we show that super-convergence provides a greater boost in\nperformance relative to standard training when the amount of labeled training\ndata is limited. The architectures and code to replicate the figures in this\npaper are available at github.com/lnsmith54/super-convergence. See\nhttp://www.fast.ai/2018/04/30/dawnbench-fastai/ for an application of\nsuper-convergence to win the DAWNBench challenge (see\nhttps://dawn.cs.stanford.edu/benchmark/).","url_abs":"http://arxiv.org/abs/1708.07120v3","url_pdf":"http://arxiv.org/pdf/1708.07120v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"super-convergence-very-fast-training-of-1","repo_url":"https://github.com/lnsmith54/super-convergence","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"caffe2","reach":{"status":"unanswered"}},{"paper_slug":"super-convergence-very-fast-training-of-1","repo_url":"https://github.com/benihime91/one_cycle_lr-tensorflow","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"super-convergence-very-fast-training-of-1","repo_url":"https://github.com/benihime91/tensorflow-on-steroids","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"super-convergence-very-fast-training-of-1","repo_url":"https://github.com/coxy1989/superconv","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"super-convergence-very-fast-training-of-1","repo_url":"https://github.com/jagatabhay/TSAI","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"super-convergence-very-fast-training-of-1","repo_url":"https://github.com/jessefokkinga/MechanismsOfAction","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"super-convergence-very-fast-training-of-1","repo_url":"https://github.com/johntd54/stanford_car","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"super-convergence-very-fast-training-of-1","repo_url":"https://github.com/scaomath/fourier-transformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"super-convergence-very-fast-training-of-1","repo_url":"https://github.com/scaomath/galerkin-transformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"super-convergence-very-fast-training-of-1","repo_url":"https://github.com/xultaeculcis/coral-net","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1708.07120","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}