{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-disciplined-approach-to-neural-network","title":"A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay","arxiv_id":"1803.09820","date":"2018-03-26","proceeding":null,"authors":["Leslie N. Smith"],"abstract":"Although deep learning has produced dazzling successes for applications of\nimage, speech, and video processing in the past few years, most trainings are\nwith suboptimal hyper-parameters, requiring unnecessarily long training times.\nSetting the hyper-parameters remains a black art that requires years of\nexperience to acquire. This report proposes several efficient ways to set the\nhyper-parameters that significantly reduce training time and improves\nperformance. Specifically, this report shows how to examine the training\nvalidation/test loss function for subtle clues of underfitting and overfitting\nand suggests guidelines for moving toward the optimal balance point. Then it\ndiscusses how to increase/decrease the learning rate/momentum to speed up\ntraining. Our experiments show that it is crucial to balance every manner of\nregularization for each dataset and architecture. Weight decay is used as a\nsample regularizer to show how its optimal value is tightly coupled with the\nlearning rates and momentums. Files to help replicate the results reported here\nare available.","url_abs":"http://arxiv.org/abs/1803.09820v2","url_pdf":"http://arxiv.org/pdf/1803.09820v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/lnsmith54/hyperParam1","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/AbhimanyuAryan/ImageClassification","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/AlexMGitHub/Checkers-MCTS","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/AmolMavuduru/CLRDeepLearning","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/DrHB/fastai_wd","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/GPUPhobia/vocal-mask","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/GilesStrong/Smith_HyperParams1_Demo","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/MatthewAlmeida/ActivityRecognitionKeras","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/Niteeshaballa/fastai-practice","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/Vakihito/SentimentYoutube","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/asvcode/1_cycle","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/bentrevett/pytorch-image-classification","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/csvance/onecycle-cosine","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/datalass1/fastai","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/ferhatkkochan/deeplearning-notes","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/fkochan/deeplearning-notes","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/gurucharanmk/PyTorch_OneCyclePolicyScheduler","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/haritha91/1cycle-Policy-Experiment","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/ifrit98/lr_range_test","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/jianshen92/stanford-car-grab-challenge","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/nachiket273/One_Cycle_Policy","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/nathanhubens/KerasOneCycle","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/pukkapies/urop2019","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/rajs96/ULMFiT-Twitter-US-Airline-Sentiment","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/rupaai/60DaysOfUdacity","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/simongrest/farm-pin-crop-detection-challenge","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/suhas1999/Flip-kart-grid-challenge","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/ymittal23/PlayWithCifar","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/MindSpore-scientific-2/code-4/tree/main/hyperparameters-part1","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"a-disciplined-approach-to-neural-network","repo_url":"https://github.com/MindSpore-scientific-2/code-9/tree/main/hyperparameters-part1","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[],"methods":[{"method_slug":"1cycle","method_name":"1cycle"},{"method_slug":"speed","method_name":"SPEED"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[{"slug":"1cycle","name":"1cycle","full_name":"1cycle learning rate scheduling policy"}],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1803.09820","atlas_url":"https://app.syntology.ai/?focus=1803.09820","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1803.09820"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pukkapies/urop2019","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/csvance/onecycle-cosine","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rupaai/60DaysOfUdacity","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gurucharanmk/PyTorch_OneCyclePolicyScheduler","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/DrHB/fastai_wd","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nachiket273/One_Cycle_Policy","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nathanhubens/KerasOneCycle","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jianshen92/stanford-car-grab-challenge","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fkochan/deeplearning-notes","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Vakihito/SentimentYoutube","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Niteeshaballa/fastai-practice","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindSpore-scientific-2/code-9/tree/main/hyperparameters-part1","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/datalass1/fastai","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ymittal23/PlayWithCifar","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/asvcode/1_cycle","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MatthewAlmeida/ActivityRecognitionKeras","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/simongrest/farm-pin-crop-detection-challenge","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AmolMavuduru/CLRDeepLearning","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rajs96/ULMFiT-Twitter-US-Airline-Sentiment","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/haritha91/1cycle-Policy-Experiment","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ifrit98/lr_range_test","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/suhas1999/Flip-kart-grid-challenge","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lnsmith54/hyperParam1","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bentrevett/pytorch-image-classification","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/GilesStrong/Smith_HyperParams1_Demo","reach":null}],"summary":{"ran_honours":1,"ran_draft_wrong":1,"ran_violates":1,"unverified":4},"by_repo_kind":{"listed":{"samples":7,"ran":3,"repositories":4}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"3dbd243a8c21052a","entry":"AMS","repo":"GilesStrong/Smith_HyperParams1_Demo","repo_kind":"listed","path":"Modules/AMS.py","file_url":"https://github.com/GilesStrong/Smith_HyperParams1_Demo/blob/HEAD/Modules/AMS.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3dbd243a8c21052a"}},{"code_sha256_prefix":"a5ca0276a41b2558","entry":"amsScanQuick","repo":"GilesStrong/Smith_HyperParams1_Demo","repo_kind":"listed","path":"Modules/AMS.py","file_url":"https://github.com/GilesStrong/Smith_HyperParams1_Demo/blob/HEAD/Modules/AMS.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a5ca0276a41b2558"}},{"code_sha256_prefix":"c12d423641e370fe","entry":"annealing_cos","repo":"csvance/onecycle-cosine","repo_kind":"listed","path":"onecyclec.py","file_url":"https://github.com/csvance/onecycle-cosine/blob/HEAD/onecyclec.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c12d423641e370fe"}},{"code_sha256_prefix":"ba3cf0e825859191","entry":"plot_history","repo":"AlexMGitHub/Checkers-MCTS","repo_kind":"listed","path":"training_pipeline.py","file_url":"https://github.com/AlexMGitHub/Checkers-MCTS/blob/HEAD/training_pipeline.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ba3cf0e825859191"}},{"code_sha256_prefix":"01862d3a9e702e3b","entry":"ploty","repo":"ifrit98/lr_range_test","repo_kind":"listed","path":"lr_range_test/plot.py","file_url":"https://github.com/ifrit98/lr_range_test/blob/HEAD/lr_range_test/plot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"01862d3a9e702e3b"}},{"code_sha256_prefix":"ef6d4cd8f9f23f77","entry":"save_nn_to_disk","repo":"AlexMGitHub/Checkers-MCTS","repo_kind":"listed","path":"training_pipeline.py","file_url":"https://github.com/AlexMGitHub/Checkers-MCTS/blob/HEAD/training_pipeline.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ef6d4cd8f9f23f77"}},{"code_sha256_prefix":"13b8695d58380dd5","entry":"states_to_piece_positions","repo":"AlexMGitHub/Checkers-MCTS","repo_kind":"listed","path":"play_Checkers.py","file_url":"https://github.com/AlexMGitHub/Checkers-MCTS/blob/HEAD/play_Checkers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"13b8695d58380dd5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}