{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/stochastic-gradient-descent-with","title":"Stochastic Gradient Descent with Preconditioned Polyak Step-size","arxiv_id":"2310.02093","date":"2023-10-03","proceeding":null,"authors":["Farshed Abdukhakimov","Chulu Xiang","Dmitry Kamzolov","Martin Takáč"],"abstract":"Stochastic Gradient Descent (SGD) is one of the many iterative optimization methods that are widely used in solving machine learning problems. These methods display valuable properties and attract researchers and industrial machine learning engineers with their simplicity. However, one of the weaknesses of this type of methods is the necessity to tune learning rate (step-size) for every loss function and dataset combination to solve an optimization problem and get an efficient performance in a given time budget. Stochastic Gradient Descent with Polyak Step-size (SPS) is a method that offers an update rule that alleviates the need of fine-tuning the learning rate of an optimizer. In this paper, we propose an extension of SPS that employs preconditioning techniques, such as Hutchinson's method, Adam, and AdaGrad, to improve its performance on badly scaled and/or ill-conditioned datasets.","url_abs":"https://arxiv.org/abs/2310.02093v1","url_pdf":"https://arxiv.org/pdf/2310.02093v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"stochastic-gradient-descent-with","repo_url":"https://github.com/fxrshed/scaledsps","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"adagrad","method_name":"AdaGrad"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"sps-1","method_name":"SPS"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.02093","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.02093"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fxrshed/scaledsps","reach":{"status":"ok"}}],"summary":{"ran":13,"unverified":2},"by_repo_kind":{"official":{"samples":15,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"cb80fce98dfd4ff2","entry":"citardouq_solve","repo":"fxrshed/scaledsps","repo_kind":"official","path":"exper/run_ps.py","file_url":"https://github.com/fxrshed/scaledsps/blob/HEAD/exper/run_ps.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb80fce98dfd4ff2"}},{"code_sha256_prefix":"7bf9abf72409d634","entry":"eval_model","repo":"fxrshed/scaledsps","repo_kind":"official","path":"run_nn.py","file_url":"https://github.com/fxrshed/scaledsps/blob/HEAD/run_nn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7bf9abf72409d634"}},{"code_sha256_prefix":"a8ea126c291f9b06","entry":"get_loss","repo":"fxrshed/scaledsps","repo_kind":"official","path":"loss_fns.py","file_url":"https://github.com/fxrshed/scaledsps/blob/HEAD/loss_fns.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a8ea126c291f9b06"}},{"code_sha256_prefix":"65d08a176b590de7","entry":"get_model","repo":"fxrshed/scaledsps","repo_kind":"official","path":"nn_models.py","file_url":"https://github.com/fxrshed/scaledsps/blob/HEAD/nn_models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"65d08a176b590de7"}},{"code_sha256_prefix":"3848a610f3f95be2","entry":"grad_logreg","repo":"fxrshed/scaledsps","repo_kind":"official","path":"new_notebooks/loss_functions.py","file_url":"https://github.com/fxrshed/scaledsps/blob/HEAD/new_notebooks/loss_functions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3848a610f3f95be2"}},{"code_sha256_prefix":"469bd75ac04bf61d","entry":"hess_logreg","repo":"fxrshed/scaledsps","repo_kind":"official","path":"new_notebooks/loss_functions.py","file_url":"https://github.com/fxrshed/scaledsps/blob/HEAD/new_notebooks/loss_functions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"469bd75ac04bf61d"}},{"code_sha256_prefix":"f518b98f3a655216","entry":"hvp_from_grad","repo":"fxrshed/scaledsps","repo_kind":"official","path":"utils.py","file_url":"https://github.com/fxrshed/scaledsps/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f518b98f3a655216"}},{"code_sha256_prefix":"745e071921d85bf8","entry":"logistic_reg","repo":"fxrshed/scaledsps","repo_kind":"official","path":"loss_fns.py","file_url":"https://github.com/fxrshed/scaledsps/blob/HEAD/loss_fns.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"745e071921d85bf8"}},{"code_sha256_prefix":"5edf07cb71dc8ea6","entry":"logreg","repo":"fxrshed/scaledsps","repo_kind":"official","path":"new_notebooks/loss_functions.py","file_url":"https://github.com/fxrshed/scaledsps/blob/HEAD/new_notebooks/loss_functions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5edf07cb71dc8ea6"}},{"code_sha256_prefix":"36c75a5c0019fe5e","entry":"nllsq","repo":"fxrshed/scaledsps","repo_kind":"official","path":"loss_fns.py","file_url":"https://github.com/fxrshed/scaledsps/blob/HEAD/loss_fns.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"36c75a5c0019fe5e"}},{"code_sha256_prefix":"892e63a09ce9b67f","entry":"rademacher","repo":"fxrshed/scaledsps","repo_kind":"official","path":"utils.py","file_url":"https://github.com/fxrshed/scaledsps/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"892e63a09ce9b67f"}},{"code_sha256_prefix":"9edee4ad4f407661","entry":"rademacher_old","repo":"fxrshed/scaledsps","repo_kind":"official","path":"exper/run_bpsps2.py","file_url":"https://github.com/fxrshed/scaledsps/blob/HEAD/exper/run_bpsps2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9edee4ad4f407661"}},{"code_sha256_prefix":"1c48a195f11c8ed0","entry":"restricted_float","repo":"fxrshed/scaledsps","repo_kind":"official","path":"utils.py","file_url":"https://github.com/fxrshed/scaledsps/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1c48a195f11c8ed0"}},{"code_sha256_prefix":"4cf659899ccad767","entry":"diag_estimate_old","repo":"fxrshed/scaledsps","repo_kind":"official","path":"exper/run_ps.py","file_url":"https://github.com/fxrshed/scaledsps/blob/HEAD/exper/run_ps.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4cf659899ccad767"}},{"code_sha256_prefix":"ef944e03deab32b1","entry":"get_dataset","repo":"fxrshed/scaledsps","repo_kind":"official","path":"datasets.py","file_url":"https://github.com/fxrshed/scaledsps/blob/HEAD/datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ef944e03deab32b1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}