{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/coupling-adaptive-batch-sizes-with-learning","title":"Coupling Adaptive Batch Sizes with Learning Rates","arxiv_id":"1612.05086","date":"2016-12-15","proceeding":null,"authors":["Lukas Balles","Javier Romero","Philipp Hennig"],"abstract":"Mini-batch stochastic gradient descent and variants thereof have become\nstandard for large-scale empirical risk minimization like the training of\nneural networks. These methods are usually used with a constant batch size\nchosen by simple empirical inspection. The batch size significantly influences\nthe behavior of the stochastic optimization algorithm, though, since it\ndetermines the variance of the gradient estimates. This variance also changes\nover the optimization process; when using a constant batch size, stability and\nconvergence is thus often enforced by means of a (manually tuned) decreasing\nlearning rate schedule.\n  We propose a practical method for dynamic batch size adaptation. It estimates\nthe variance of the stochastic gradients and adapts the batch size to decrease\nthe variance proportionally to the value of the objective function, removing\nthe need for the aforementioned learning rate decrease. In contrast to recent\nrelated work, our algorithm couples the batch size to the learning rate,\ndirectly reflecting the known relationship between the two. On popular image\nclassification benchmarks, our batch size adaptation yields faster optimization\nconvergence, while simultaneously simplifying learning rate tuning. A\nTensorFlow implementation is available.","url_abs":"http://arxiv.org/abs/1612.05086v2","url_pdf":"http://arxiv.org/pdf/1612.05086v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"coupling-adaptive-batch-sizes-with-learning","repo_url":"https://github.com/ProbabilisticNumerics/cabs","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"stochastic-optimization","task_name":"Stochastic Optimization"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1612.05086","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1612.05086"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ProbabilisticNumerics/cabs","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":1},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"897a48a2ca37ef0c","entry":"grads_and_grad_moms","repo":"ProbabilisticNumerics/cabs","repo_kind":"official","path":"gradient_moment.py","file_url":"https://github.com/ProbabilisticNumerics/cabs/blob/HEAD/gradient_moment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"897a48a2ca37ef0c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}