{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/nag-gs-semi-implicit-accelerated-and-robust","title":"NAG-GS: Semi-Implicit, Accelerated and Robust Stochastic Optimizer","arxiv_id":"2209.14937","date":"2022-09-29","proceeding":null,"authors":["Valentin Leplat","Daniil Merkulov","Aleksandr Katrutsa","Daniel Bershatsky","Olga Tsymboi","Ivan Oseledets"],"abstract":"Classical machine learning models such as deep neural networks are usually trained by using Stochastic Gradient Descent-based (SGD) algorithms. The classical SGD can be interpreted as a discretization of the stochastic gradient flow. In this paper we propose a novel, robust and accelerated stochastic optimizer that relies on two key elements: (1) an accelerated Nesterov-like Stochastic Differential Equation (SDE) and (2) its semi-implicit Gauss-Seidel type discretization. The convergence and stability of the obtained method, referred to as NAG-GS, are first studied extensively in the case of the minimization of a quadratic function. This analysis allows us to come up with an optimal learning rate in terms of the convergence rate while ensuring the stability of NAG-GS. This is achieved by the careful analysis of the spectral radius of the iteration matrix and the covariance matrix at stationarity with respect to all hyperparameters of our method. Further, we show that NAG- GS is competitive with state-of-the-art methods such as momentum SGD with weight decay and AdamW for the training of machine learning models such as the logistic regression model, the residual networks models on standard computer vision datasets, Transformers in the frame of the GLUE benchmark and the recent Vision Transformers.","url_abs":"https://arxiv.org/abs/2209.14937v2","url_pdf":"https://arxiv.org/pdf/2209.14937v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"nag-gs-semi-implicit-accelerated-and-robust","repo_url":"https://github.com/skolai/nag-gs","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"nag-gs-semi-implicit-accelerated-and-robust","repo_url":"https://github.com/naggsopt/naggs","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"adamw","method_name":"AdamW"},{"method_slug":"logistic-regression","method_name":"Logistic Regression"},{"method_slug":"sgd","method_name":"SGD"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2209.14937","atlas_url":"https://app.syntology.ai/?focus=2209.14937","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2209.14937"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/naggsopt/naggs","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/skolai/nag-gs","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":2,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1407df1285645f84","entry":"nag_gs","repo":"skolai/nag-gs","repo_kind":"official","path":"nag_gs/nag_gs_pytorch.py","file_url":"https://github.com/skolai/nag-gs/blob/HEAD/nag_gs/nag_gs_pytorch.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1407df1285645f84"}},{"code_sha256_prefix":"cfd097e4d8975ee6","entry":"scale_by_nag4","repo":"skolai/nag-gs","repo_kind":"official","path":"nag_gs/nag_gs_jax.py","file_url":"https://github.com/skolai/nag-gs/blob/HEAD/nag_gs/nag_gs_jax.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cfd097e4d8975ee6"}},{"code_sha256_prefix":"dd9eb5038965744f","entry":"nag4","repo":"skolai/nag-gs","repo_kind":"official","path":"nag_gs/nag_gs_jax.py","file_url":"https://github.com/skolai/nag-gs/blob/HEAD/nag_gs/nag_gs_jax.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dd9eb5038965744f"}},{"code_sha256_prefix":"fe293fce575a5997","entry":"update_trace","repo":"skolai/nag-gs","repo_kind":"official","path":"nag_gs/nag_gs_jax.py","file_url":"https://github.com/skolai/nag-gs/blob/HEAD/nag_gs/nag_gs_jax.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fe293fce575a5997"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}