{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/why-warmup-the-learning-rate-underlying","title":"Why Warmup the Learning Rate? Underlying Mechanisms and Improvements","arxiv_id":"2406.09405","date":"2024-06-13","proceeding":null,"authors":["Dayal Singh Kalra","Maissam Barkeshli"],"abstract":"It is common in deep learning to warm up the learning rate $\\eta$, often by a linear schedule between $\\eta_{\\text{init}} = 0$ and a predetermined target $\\eta_{\\text{trgt}}$. In this paper, we show through systematic experiments using SGD and Adam that the overwhelming benefit of warmup arises from allowing the network to tolerate larger $\\eta_{\\text{trgt}}$ {by forcing the network to more well-conditioned areas of the loss landscape}. The ability to handle larger $\\eta_{\\text{trgt}}$ makes hyperparameter tuning more robust while improving the final performance. We uncover different regimes of operation during the warmup period, depending on whether training starts off in a progressive sharpening or sharpness reduction phase, which in turn depends on the initialization and parameterization. Using these insights, we show how $\\eta_{\\text{init}}$ can be properly chosen by utilizing the loss catapult mechanism, which saves on the number of warmup steps, in some cases completely eliminating the need for warmup. We also suggest an initialization for the variance in Adam which provides benefits similar to warmup.","url_abs":"https://arxiv.org/abs/2406.09405v2","url_pdf":"https://arxiv.org/pdf/2406.09405v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.09405","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.09405"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/dayal-kalra/why-warmup","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"26d13d6c867ba7fa","entry":"cross_entropy_loss","repo":"dayal-kalra/why-warmup","repo_kind":"found_in_text","path":"vision/mechanisms/utils/loss_utils.py","file_url":"https://github.com/dayal-kalra/why-warmup/blob/HEAD/vision/mechanisms/utils/loss_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"26d13d6c867ba7fa"}},{"code_sha256_prefix":"b9ac929ad3f0fc94","entry":"cross_entropy_loss_integer_labels","repo":"dayal-kalra/why-warmup","repo_kind":"found_in_text","path":"vision/mechanisms/utils/loss_utils.py","file_url":"https://github.com/dayal-kalra/why-warmup/blob/HEAD/vision/mechanisms/utils/loss_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b9ac929ad3f0fc94"}},{"code_sha256_prefix":"facc696d7f68e5d0","entry":"linear_warmup","repo":"dayal-kalra/why-warmup","repo_kind":"found_in_text","path":"vision/mechanisms/train_resnet_grads_adam_lobpcg.py","file_url":"https://github.com/dayal-kalra/why-warmup/blob/HEAD/vision/mechanisms/train_resnet_grads_adam_lobpcg.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"facc696d7f68e5d0"}},{"code_sha256_prefix":"8294c2b79bb5a70f","entry":"mse_loss","repo":"dayal-kalra/why-warmup","repo_kind":"found_in_text","path":"vision/mechanisms/utils/loss_utils.py","file_url":"https://github.com/dayal-kalra/why-warmup/blob/HEAD/vision/mechanisms/utils/loss_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8294c2b79bb5a70f"}},{"code_sha256_prefix":"5cdd8100958da566","entry":"count_parameters","repo":"dayal-kalra/why-warmup","repo_kind":"found_in_text","path":"vision/mechanisms/utils/model_utils.py","file_url":"https://github.com/dayal-kalra/why-warmup/blob/HEAD/vision/mechanisms/utils/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5cdd8100958da566"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}