{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sania-polyak-type-optimization-framework","title":"SANIA: Polyak-type Optimization Framework Leads to Scale Invariant Stochastic Algorithms","arxiv_id":"2312.17369","date":"2023-12-28","proceeding":null,"authors":["Farshed Abdukhakimov","Chulu Xiang","Dmitry Kamzolov","Robert Gower","Martin Takáč"],"abstract":"Adaptive optimization methods are widely recognized as among the most popular approaches for training Deep Neural Networks (DNNs). Techniques such as Adam, AdaGrad, and AdaHessian utilize a preconditioner that modifies the search direction by incorporating information about the curvature of the objective function. However, despite their adaptive characteristics, these methods still require manual fine-tuning of the step-size. This, in turn, impacts the time required to solve a particular problem. This paper presents an optimization framework named SANIA to tackle these challenges. Beyond eliminating the need for manual step-size hyperparameter settings, SANIA incorporates techniques to address poorly scaled or ill-conditioned problems. We also explore several preconditioning methods, including Hutchinson's method, which approximates the Hessian diagonal of the loss function. We conclude with an extensive empirical examination of the proposed techniques across classification tasks, covering both convex and non-convex contexts.","url_abs":"https://arxiv.org/abs/2312.17369v1","url_pdf":"https://arxiv.org/pdf/2312.17369v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sania-polyak-type-optimization-framework","repo_url":"https://github.com/fxrshed/sania","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"adagrad","method_name":"AdaGrad"},{"method_slug":"adahessian","method_name":"AdaHessian"},{"method_slug":"adam","method_name":"Adam"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.17369","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.17369"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fxrshed/sania","reach":{"status":"ok"}}],"summary":{"ran":10},"by_repo_kind":{"official":{"samples":10,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"3848a610f3f95be2","entry":"grad_logreg","repo":"fxrshed/sania","repo_kind":"official","path":"experiments/loss_functions.py","file_url":"https://github.com/fxrshed/sania/blob/HEAD/experiments/loss_functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3848a610f3f95be2"}},{"code_sha256_prefix":"469bd75ac04bf61d","entry":"hess_logreg","repo":"fxrshed/sania","repo_kind":"official","path":"experiments/loss_functions.py","file_url":"https://github.com/fxrshed/sania/blob/HEAD/experiments/loss_functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"469bd75ac04bf61d"}},{"code_sha256_prefix":"5edf07cb71dc8ea6","entry":"logreg","repo":"fxrshed/sania","repo_kind":"official","path":"experiments/loss_functions.py","file_url":"https://github.com/fxrshed/sania/blob/HEAD/experiments/loss_functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5edf07cb71dc8ea6"}},{"code_sha256_prefix":"ac62432dc5134b0d","entry":"make_layers","repo":"fxrshed/sania","repo_kind":"official","path":"experiments/models/vgg.py","file_url":"https://github.com/fxrshed/sania/blob/HEAD/experiments/models/vgg.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ac62432dc5134b0d"}},{"code_sha256_prefix":"ff484a1ce7b2b244","entry":"resnet20","repo":"fxrshed/sania","repo_kind":"official","path":"experiments/cifar_resnet.py","file_url":"https://github.com/fxrshed/sania/blob/HEAD/experiments/cifar_resnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ff484a1ce7b2b244"}},{"code_sha256_prefix":"472997c84869ac7f","entry":"resnet32","repo":"fxrshed/sania","repo_kind":"official","path":"experiments/cifar_resnet.py","file_url":"https://github.com/fxrshed/sania/blob/HEAD/experiments/cifar_resnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"472997c84869ac7f"}},{"code_sha256_prefix":"65164998e66af8ac","entry":"resnet44","repo":"fxrshed/sania","repo_kind":"official","path":"experiments/cifar_resnet.py","file_url":"https://github.com/fxrshed/sania/blob/HEAD/experiments/cifar_resnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"65164998e66af8ac"}},{"code_sha256_prefix":"e3f49bbe6b2927c0","entry":"run_optimizer","repo":"fxrshed/sania","repo_kind":"official","path":"experiments/_train_cifar10.py","file_url":"https://github.com/fxrshed/sania/blob/HEAD/experiments/_train_cifar10.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e3f49bbe6b2927c0"}},{"code_sha256_prefix":"a5175f93f3287752","entry":"sania_adagrad","repo":"fxrshed/sania","repo_kind":"official","path":"experiments/_train_cifar10.py","file_url":"https://github.com/fxrshed/sania/blob/HEAD/experiments/_train_cifar10.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a5175f93f3287752"}},{"code_sha256_prefix":"7207a8b483d0d70f","entry":"sania_adam","repo":"fxrshed/sania","repo_kind":"official","path":"experiments/_train_cifar10.py","file_url":"https://github.com/fxrshed/sania/blob/HEAD/experiments/_train_cifar10.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7207a8b483d0d70f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}