{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/variance-reduced-clipping-for-non-convex","title":"Variance-reduced Clipping for Non-convex Optimization","arxiv_id":"2303.00883","date":"2023-03-02","proceeding":null,"authors":["Amirhossein Reisizadeh","Haochuan Li","Subhro Das","Ali Jadbabaie"],"abstract":"Gradient clipping is a standard training technique used in deep learning applications such as large-scale language modeling to mitigate exploding gradients. Recent experimental studies have demonstrated a fairly special behavior in the smoothness of the training objective along its trajectory when trained with gradient clipping. That is, the smoothness grows with the gradient norm. This is in clear contrast to the well-established assumption in folklore non-convex optimization, a.k.a. $L$--smoothness, where the smoothness is assumed to be bounded by a constant $L$ globally. The recently introduced $(L_0,L_1)$--smoothness is a more relaxed notion that captures such behavior in non-convex optimization. In particular, it has been shown that under this relaxed smoothness assumption, SGD with clipping requires $O(\\epsilon^{-4})$ stochastic gradient computations to find an $\\epsilon$--stationary solution. In this paper, we employ a variance reduction technique, namely SPIDER, and demonstrate that for a carefully designed learning rate, this complexity is improved to $O(\\epsilon^{-3})$ which is order-optimal. Our designed learning rate comprises the clipping technique to mitigate the growing smoothness. Moreover, when the objective function is the average of $n$ components, we improve the existing $O(n\\epsilon^{-2})$ bound on the stochastic gradient complexity to $O(\\sqrt{n} \\epsilon^{-2} + n)$, which is order-optimal as well. In addition to being theoretically optimal, SPIDER with our designed parameters demonstrates comparable empirical performance against variance-reduced methods such as SVRG and SARAH in several vision tasks.","url_abs":"https://arxiv.org/abs/2303.00883v2","url_pdf":"https://arxiv.org/pdf/2303.00883v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"variance-reduced-clipping-for-non-convex","repo_url":"https://github.com/haochuan-mit/varaince-reduced-clipping-for-non-convex-optimization","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2303.00883","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.00883"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/haochuan-mit/varaince-reduced-clipping-for-non-convex-optimization","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"56627193cdc83561","entry":"fcn","repo":"haochuan-mit/varaince-reduced-clipping-for-non-convex-optimization","repo_kind":"official","path":"vr_optimizers/utils/models/fcn_mnist.py","file_url":"https://github.com/haochuan-mit/varaince-reduced-clipping-for-non-convex-optimization/blob/HEAD/vr_optimizers/utils/models/fcn_mnist.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"56627193cdc83561"}},{"code_sha256_prefix":"4291fa670f696748","entry":"get_grad_norm","repo":"haochuan-mit/varaince-reduced-clipping-for-non-convex-optimization","repo_kind":"official","path":"vr_optimizers/optims/utils.py","file_url":"https://github.com/haochuan-mit/varaince-reduced-clipping-for-non-convex-optimization/blob/HEAD/vr_optimizers/optims/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4291fa670f696748"}},{"code_sha256_prefix":"eb67c89e0a810e92","entry":"initialise_arg_parser","repo":"haochuan-mit/varaince-reduced-clipping-for-non-convex-optimization","repo_kind":"official","path":"vr_optimizers/opts.py","file_url":"https://github.com/haochuan-mit/varaince-reduced-clipping-for-non-convex-optimization/blob/HEAD/vr_optimizers/opts.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eb67c89e0a810e92"}},{"code_sha256_prefix":"551b0a8400d30f9f","entry":"parse_args","repo":"haochuan-mit/varaince-reduced-clipping-for-non-convex-optimization","repo_kind":"official","path":"vr_optimizers/opts.py","file_url":"https://github.com/haochuan-mit/varaince-reduced-clipping-for-non-convex-optimization/blob/HEAD/vr_optimizers/opts.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"551b0a8400d30f9f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}