{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/chaotic-regularization-and-heavy-tailed","title":"Chaotic Regularization and Heavy-Tailed Limits for Deterministic Gradient Descent","arxiv_id":"2205.11361","date":"2022-05-23","proceeding":null,"authors":["Soon Hoe Lim","Yijun Wan","Umut Şimşekli"],"abstract":"Recent studies have shown that gradient descent (GD) can achieve improved generalization when its dynamics exhibits a chaotic behavior. However, to obtain the desired effect, the step-size should be chosen sufficiently large, a task which is problem dependent and can be difficult in practice. In this study, we incorporate a chaotic component to GD in a controlled manner, and introduce multiscale perturbed GD (MPGD), a novel optimization framework where the GD recursion is augmented with chaotic perturbations that evolve via an independent dynamical system. We analyze MPGD from three different angles: (i) By building up on recent advances in rough paths theory, we show that, under appropriate assumptions, as the step-size decreases, the MPGD recursion converges weakly to a stochastic differential equation (SDE) driven by a heavy-tailed L\\'evy-stable process. (ii) By making connections to recently developed generalization bounds for heavy-tailed processes, we derive a generalization bound for the limiting SDE and relate the worst-case generalization error over the trajectories of the process to the parameters of MPGD. (iii) We analyze the implicit regularization effect brought by the dynamical regularization and show that, in the weak perturbation regime, MPGD introduces terms that penalize the Hessian of the loss function. Empirical results are provided to demonstrate the advantages of MPGD.","url_abs":"https://arxiv.org/abs/2205.11361v2","url_pdf":"https://arxiv.org/pdf/2205.11361v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"chaotic-regularization-and-heavy-tailed","repo_url":"https://github.com/shoelim/mpgd","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"generalization-bounds","task_name":"Generalization Bounds"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2205.11361","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2205.11361"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/JonasGeiping/fullbatchtraining","reach":{"status":"ok","spdx":"LGPL-2.1"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shoelim/mpgd","reach":null}],"summary":{"ran":1,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"dac249a587c6c782","entry":"SGD_AGC","repo":"shoelim/mpgd","repo_kind":"official","path":"cifar10_classification/fullbatch/training/additional_optimizers/sgd_agc.py","file_url":"https://github.com/shoelim/mpgd/blob/HEAD/cifar10_classification/fullbatch/training/additional_optimizers/sgd_agc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"dac249a587c6c782"}},{"code_sha256_prefix":"35a86fdf1f6b3a48","entry":"unitwise_norm","repo":"shoelim/mpgd","repo_kind":"official","path":"cifar10_classification/fullbatch/training/additional_optimizers/sgd_agc.py","file_url":"https://github.com/shoelim/mpgd/blob/HEAD/cifar10_classification/fullbatch/training/additional_optimizers/sgd_agc.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"35a86fdf1f6b3a48"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}