{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/online-batch-selection-for-faster-training-of","title":"Online Batch Selection for Faster Training of Neural Networks","arxiv_id":"1511.06343","date":"2015-11-19","proceeding":null,"authors":["Ilya Loshchilov","Frank Hutter"],"abstract":"Deep neural networks are commonly trained using stochastic non-convex\noptimization procedures, which are driven by gradient information estimated on\nfractions (batches) of the dataset. While it is commonly accepted that batch\nsize is an important parameter for offline tuning, the benefits of online\nselection of batches remain poorly understood. We investigate online batch\nselection strategies for two state-of-the-art methods of stochastic\ngradient-based optimization, AdaDelta and Adam. As the loss function to be\nminimized for the whole dataset is an aggregation of loss functions of\nindividual datapoints, intuitively, datapoints with the greatest loss should be\nconsidered (selected in a batch) more frequently. However, the limitations of\nthis intuition and the proper control of the selection pressure over time are\nopen questions. We propose a simple strategy where all datapoints are ranked\nw.r.t. their latest known loss value and the probability to be selected decays\nexponentially as a function of rank. Our experimental results on the MNIST\ndataset suggest that selecting batches speeds up both AdaDelta and Adam by a\nfactor of about 5.","url_abs":"http://arxiv.org/abs/1511.06343v4","url_pdf":"http://arxiv.org/pdf/1511.06343v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"online-batch-selection-for-faster-training-of","repo_url":"https://github.com/Lasagne/Lasagne","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[],"methods":[{"method_slug":"adadelta","method_name":"AdaDelta"},{"method_slug":"adam","method_name":"Adam"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1511.06343","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}