{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/automated-learning-rate-scheduler-for-large","title":"Automated Learning Rate Scheduler for Large-batch Training","arxiv_id":"2107.05855","date":"2021-07-13","proceeding":"ICML Workshop AutoML 2021 7","authors":["Chiheon Kim","Saehoon Kim","Jongmin Kim","Donghoon Lee","Sungwoong Kim"],"abstract":"Large-batch training has been essential in leveraging large-scale datasets and models in deep learning. While it is computationally beneficial to use large batch sizes, it often requires a specially designed learning rate (LR) schedule to achieve a comparable level of performance as in smaller batch training. Especially, when the number of training epochs is constrained, the use of a large LR and a warmup strategy is critical in the final performance of large-batch training due to the reduced number of updating steps. In this work, we propose an automated LR scheduling algorithm which is effective for neural network training with a large batch size under the given epoch budget. In specific, the whole schedule consists of two phases: adaptive warmup and predefined decay, where the LR is increased until the training loss no longer decreases and decreased to zero until the end of training. Here, whether the training loss has reached the minimum value is robustly checked with Gaussian process smoothing in an online manner with a low computational burden. Coupled with adaptive stochastic optimizers such as AdamP and LAMB, the proposed scheduler successfully adjusts the LRs without cumbersome hyperparameter tuning and achieves comparable or better performances than tuned baselines on various image classification benchmarks and architectures with a wide range of batch sizes.","url_abs":"https://arxiv.org/abs/2107.05855v1","url_pdf":"https://arxiv.org/pdf/2107.05855v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"automated-learning-rate-scheduler-for-large","repo_url":"https://github.com/kakaobrain/autowu","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"scheduling","task_name":"Scheduling"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"gaussian-process","method_name":"Gaussian Process"},{"method_slug":"lamb","method_name":"LAMB"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2107.05855","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2107.05855"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/kakaobrain/autowu","reach":null}],"summary":{"ran":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"88433b08b053e5d3","entry":"AutoWU","repo":"kakaobrain/autowu","repo_kind":"official","path":"autowu.py","file_url":"https://github.com/kakaobrain/autowu/blob/HEAD/autowu.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"88433b08b053e5d3"}},{"code_sha256_prefix":"deb53ca3b74bec8e","entry":"CustomGP","repo":"kakaobrain/autowu","repo_kind":"official","path":"autowu.py","file_url":"https://github.com/kakaobrain/autowu/blob/HEAD/autowu.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"deb53ca3b74bec8e"}},{"code_sha256_prefix":"d83e715fb85e9189","entry":"Regressor","repo":"kakaobrain/autowu","repo_kind":"official","path":"autowu.py","file_url":"https://github.com/kakaobrain/autowu/blob/HEAD/autowu.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d83e715fb85e9189"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}