{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-view-of-mini-batch-sgd-via-generating","title":"A view of mini-batch SGD via generating functions: conditions of convergence, phase transitions, benefit from negative momenta","arxiv_id":"2206.11124","date":"2022-06-22","proceeding":null,"authors":["Maksim Velikanov","Denis Kuznedelev","Dmitry Yarotsky"],"abstract":"Mini-batch SGD with momentum is a fundamental algorithm for learning large predictive models. In this paper we develop a new analytic framework to analyze noise-averaged properties of mini-batch SGD for linear models at constant learning rates, momenta and sizes of batches. Our key idea is to consider the dynamics of the second moments of model parameters for a special family of \"Spectrally Expressible\" approximations. This allows to obtain an explicit expression for the generating function of the sequence of loss values. By analyzing this generating function, we find, in particular, that 1) the SGD dynamics exhibits several convergent and divergent regimes depending on the spectral distributions of the problem; 2) the convergent regimes admit explicit stability conditions, and explicit loss asymptotics in the case of power-law spectral distributions; 3) the optimal convergence rate can be achieved at negative momenta. We verify our theoretical predictions by extensive experiments with MNIST, CIFAR10 and synthetic problems, and find a good quantitative agreement.","url_abs":"https://arxiv.org/abs/2206.11124v2","url_pdf":"https://arxiv.org/pdf/2206.11124v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-view-of-mini-batch-sgd-via-generating","repo_url":"https://github.com/godofnothing/powerlawoptimization","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"sgd","method_name":"SGD"},{"method_slug":"sgd-with-momentum","method_name":"SGD with Momentum"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2206.11124","atlas_url":"https://app.syntology.ai/?focus=2206.11124","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.11124"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/godofnothing/powerlawoptimization","reach":null}],"summary":{"ran_fixture":1,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bda414040e88cc6c","entry":"empirical_ntk","repo":"godofnothing/powerlawoptimization","repo_kind":"official","path":"train_4_regimes.py","file_url":"https://github.com/godofnothing/powerlawoptimization/blob/HEAD/train_4_regimes.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bda414040e88cc6c"}},{"code_sha256_prefix":"9e17f30e31ba5db2","entry":"format_params","repo":"godofnothing/powerlawoptimization","repo_kind":"official","path":"train_4_regimes.py","file_url":"https://github.com/godofnothing/powerlawoptimization/blob/HEAD/train_4_regimes.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9e17f30e31ba5db2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}