{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/yellowfin-and-the-art-of-momentum-tuning","title":"YellowFin and the Art of Momentum Tuning","arxiv_id":"1706.03471","date":"2017-06-12","proceeding":"ICLR 2018 1","authors":["Jian Zhang","Ioannis Mitliagkas"],"abstract":"Hyperparameter tuning is one of the most time-consuming workloads in deep\nlearning. State-of-the-art optimizers, such as AdaGrad, RMSProp and Adam,\nreduce this labor by adaptively tuning an individual learning rate for each\nvariable. Recently researchers have shown renewed interest in simpler methods\nlike momentum SGD as they may yield better test metrics. Motivated by this\ntrend, we ask: can simple adaptive methods based on SGD perform as well or\nbetter? We revisit the momentum SGD algorithm and show that hand-tuning a\nsingle learning rate and momentum makes it competitive with Adam. We then\nanalyze its robustness to learning rate misspecification and objective\ncurvature variation. Based on these insights, we design YellowFin, an automatic\ntuner for momentum and learning rate in SGD. YellowFin optionally uses a\nnegative-feedback loop to compensate for the momentum dynamics in asynchronous\nsettings on the fly. We empirically show that YellowFin can converge in fewer\niterations than Adam on ResNets and LSTMs for image recognition, language\nmodeling and constituency parsing, with a speedup of up to 3.28x in synchronous\nand up to 2.69x in asynchronous settings.","url_abs":"http://arxiv.org/abs/1706.03471v2","url_pdf":"http://arxiv.org/pdf/1706.03471v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"yellowfin-and-the-art-of-momentum-tuning","repo_url":"https://github.com/JianGoForIt/YellowFin","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"yellowfin-and-the-art-of-momentum-tuning","repo_url":"https://github.com/JianGoForIt/YellowFin_Pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"constituency-parsing","task_name":"Constituency Parsing"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"adagrad","method_name":"AdaGrad"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"rmsprop","method_name":"RMSProp"},{"method_slug":"sgd","method_name":"SGD"},{"method_slug":"yellowfin","method_name":"YellowFin"}],"datasets_introduced":[],"methods_introduced":[{"slug":"yellowfin","name":"YellowFin","full_name":"YellowFin"}],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1706.03471","atlas_url":"https://app.syntology.ai/?focus=1706.03471","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1706.03471"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/JianGoForIt/YellowFin_Pytorch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/JianGoForIt/YellowFin","reach":null}],"summary":{"unverified":1},"by_repo_kind":{"listed":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"34a1343d41b87f8f","entry":"run_epoch","repo":"JianGoForIt/YellowFin","repo_kind":"listed","path":"ptb/model/ptb_word_lm.py","file_url":"https://github.com/JianGoForIt/YellowFin/blob/HEAD/ptb/model/ptb_word_lm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"34a1343d41b87f8f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}