{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/online-learning-rate-adaptation-with","title":"Online Learning Rate Adaptation with Hypergradient Descent","arxiv_id":"1703.04782","date":"2017-03-14","proceeding":"ICLR 2018 1","authors":["Atilim Gunes Baydin","Robert Cornish","David Martinez Rubio","Mark Schmidt","Frank Wood"],"abstract":"We introduce a general method for improving the convergence rate of\ngradient-based optimizers that is easy to implement and works well in practice.\nWe demonstrate the effectiveness of the method in a range of optimization\nproblems by applying it to stochastic gradient descent, stochastic gradient\ndescent with Nesterov momentum, and Adam, showing that it significantly reduces\nthe need for the manual tuning of the initial learning rate for these commonly\nused algorithms. Our method works by dynamically updating the learning rate\nduring optimization using the gradient with respect to the learning rate of the\nupdate rule itself. Computing this \"hypergradient\" needs little additional\ncomputation, requires only one extra copy of the original gradient to be stored\nin memory, and relies upon nothing more than what is provided by reverse-mode\nautomatic differentiation.","url_abs":"http://arxiv.org/abs/1703.04782v3","url_pdf":"http://arxiv.org/pdf/1703.04782v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"online-learning-rate-adaptation-with","repo_url":"https://github.com/gbaydin/hypergradient-descent","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"online-learning-rate-adaptation-with","repo_url":"https://github.com/awslabs/adatune","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"online-learning-rate-adaptation-with","repo_url":"https://github.com/kayuksel/pytorch-adamaio","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"hyperparameter-optimization","task_name":"Hyperparameter Optimization"},{"task_slug":"stochastic-optimization","task_name":"Stochastic Optimization"}],"methods":[{"method_slug":"adam","method_name":"Adam"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1703.04782","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1703.04782"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/awslabs/adatune","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gbaydin/hypergradient-descent","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kayuksel/pytorch-adamaio","reach":null}],"summary":{"ran":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ac62432dc5134b0d","entry":"make_layers","repo":"gbaydin/hypergradient-descent","repo_kind":"official","path":"vgg.py","file_url":"https://github.com/gbaydin/hypergradient-descent/blob/HEAD/vgg.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ac62432dc5134b0d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}