{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-with-random-learning-rates","title":"Learning with Random Learning Rates","arxiv_id":"1810.01322","date":"2018-10-02","proceeding":null,"authors":["Léonard Blier","Pierre Wolinski","Yann Ollivier"],"abstract":"Hyperparameter tuning is a bothersome step in the training of deep learning\nmodels. One of the most sensitive hyperparameters is the learning rate of the\ngradient descent. We present the 'All Learning Rates At Once' (Alrao)\noptimization method for neural networks: each unit or feature in the network\ngets its own learning rate sampled from a random distribution spanning several\norders of magnitude. This comes at practically no computational cost. Perhaps\nsurprisingly, stochastic gradient descent (SGD) with Alrao performs close to\nSGD with an optimally tuned learning rate, for various architectures and\nproblems. Alrao could save time when testing deep learning models: a range of\nmodels could be quickly assessed with Alrao, and the most promising models\ncould then be trained more extensively. This text comes with a PyTorch\nimplementation of the method, which can be plugged on an existing PyTorch\nmodel: https://github.com/leonardblier/alrao .","url_abs":"http://arxiv.org/abs/1810.01322v3","url_pdf":"http://arxiv.org/pdf/1810.01322v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-with-random-learning-rates","repo_url":"https://github.com/leonardblier/alrao","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}