{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/understanding-short-horizon-bias-in","title":"Understanding Short-Horizon Bias in Stochastic Meta-Optimization","arxiv_id":"1803.02021","date":"2018-03-06","proceeding":"ICLR 2018 1","authors":["Yuhuai Wu","Mengye Ren","Renjie Liao","Roger Grosse"],"abstract":"Careful tuning of the learning rate, or even schedules thereof, can be\ncrucial to effective neural net training. There has been much recent interest\nin gradient-based meta-optimization, where one tunes hyperparameters, or even\nlearns an optimizer, in order to minimize the expected loss when the training\nprocedure is unrolled. But because the training procedure must be unrolled\nthousands of times, the meta-objective must be defined with an\norders-of-magnitude shorter time horizon than is typical for neural net\ntraining. We show that such short-horizon meta-objectives cause a serious bias\ntowards small step sizes, an effect we term short-horizon bias. We introduce a\ntoy problem, a noisy quadratic cost function, on which we analyze short-horizon\nbias by deriving and comparing the optimal schedules for short and long time\nhorizons. We then run meta-optimization experiments (both offline and online)\non standard benchmark datasets, showing that meta-optimization chooses too\nsmall a learning rate by multiple orders of magnitude, even when run with a\nmoderately long time horizon (100 steps) typical of work in the area. We\nbelieve short-horizon bias is a fundamental problem that needs to be addressed\nif meta-optimization is to scale to practical neural net training regimes.","url_abs":"http://arxiv.org/abs/1803.02021v1","url_pdf":"http://arxiv.org/pdf/1803.02021v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"understanding-short-horizon-bias-in","repo_url":"https://github.com/renmengye/meta-optim-public","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1803.02021","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1803.02021"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/renmengye/meta-optim-public","reach":null}],"summary":{"ran_honours":1,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ada29593c0b204aa","entry":"open_folder","repo":"renmengye/meta-optim-public","repo_kind":"official","path":"online.py","file_url":"https://github.com/renmengye/meta-optim-public/blob/HEAD/online.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ada29593c0b204aa"}},{"code_sha256_prefix":"72d8c841992295dd","entry":"prepare_surface","repo":"renmengye/meta-optim-public","repo_kind":"official","path":"offline.py","file_url":"https://github.com/renmengye/meta-optim-public/blob/HEAD/offline.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"72d8c841992295dd"}},{"code_sha256_prefix":"e5f781588098ef96","entry":"plot_report_figure_combined","repo":"renmengye/meta-optim-public","repo_kind":"official","path":"online.py","file_url":"https://github.com/renmengye/meta-optim-public/blob/HEAD/online.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e5f781588098ef96"}},{"code_sha256_prefix":"dd618018a43c71a6","entry":"plot_surface","repo":"renmengye/meta-optim-public","repo_kind":"official","path":"offline.py","file_url":"https://github.com/renmengye/meta-optim-public/blob/HEAD/offline.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dd618018a43c71a6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}