{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/metagrad-multiple-learning-rates-in-online","title":"MetaGrad: Multiple Learning Rates in Online Learning","arxiv_id":"1604.08740","date":"2016-04-29","proceeding":"NeurIPS 2016 12","authors":["Tim van Erven","Wouter M. Koolen"],"abstract":"In online convex optimization it is well known that certain subclasses of\nobjective functions are much easier than arbitrary convex functions. We are\ninterested in designing adaptive methods that can automatically get fast rates\nin as many such subclasses as possible, without any manual tuning. Previous\nadaptive methods are able to interpolate between strongly convex and general\nconvex functions. We present a new method, MetaGrad, that adapts to a much\nbroader class of functions, including exp-concave and strongly convex\nfunctions, but also various types of stochastic and non-stochastic functions\nwithout any curvature. For instance, MetaGrad can achieve logarithmic regret on\nthe unregularized hinge loss, even though it has no curvature, if the data come\nfrom a favourable probability distribution. MetaGrad's main feature is that it\nsimultaneously considers multiple learning rates. Unlike previous methods with\nprovable regret guarantees, however, its learning rates are not monotonically\ndecreasing over time and are not tuned based on a theoretically derived bound\non the regret. Instead, they are weighted directly proportional to their\nempirical performance on the data using a tilted exponential weights master\nalgorithm.","url_abs":"http://arxiv.org/abs/1604.08740v3","url_pdf":"http://arxiv.org/pdf/1604.08740v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"metagrad-multiple-learning-rates-in-online","repo_url":"https://bitbucket.org/wmkoolen/metagrad","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1604.08740","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}