{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/preconditioned-stochastic-gradient-descent","title":"Preconditioned Stochastic Gradient Descent","arxiv_id":"1512.04202","date":"2015-12-14","proceeding":null,"authors":["Xi-Lin Li"],"abstract":"Stochastic gradient descent (SGD) still is the workhorse for many practical\nproblems. However, it converges slow, and can be difficult to tune. It is\npossible to precondition SGD to accelerate its convergence remarkably. But many\nattempts in this direction either aim at solving specialized problems, or\nresult in significantly more complicated methods than SGD. This paper proposes\na new method to estimate a preconditioner such that the amplitudes of\nperturbations of preconditioned stochastic gradient match that of the\nperturbations of parameters to be optimized in a way comparable to Newton\nmethod for deterministic optimization. Unlike the preconditioners based on\nsecant equation fitting as done in deterministic quasi-Newton methods, which\nassume positive definite Hessian and approximate its inverse, the new\npreconditioner works equally well for both convex and non-convex optimizations\nwith exact or noisy gradients. When stochastic gradient is used, it can\nnaturally damp the gradient noise to stabilize SGD. Efficient preconditioner\nestimation methods are developed, and with reasonable simplifications, they are\napplicable to large scaled problems. Experimental results demonstrate that\nequipped with the new preconditioner, without any tuning effort, preconditioned\nSGD can efficiently solve many challenging problems like the training of a deep\nneural network or a recurrent neural network requiring extremely long term\nmemories.","url_abs":"http://arxiv.org/abs/1512.04202v3","url_pdf":"http://arxiv.org/pdf/1512.04202v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"preconditioned-stochastic-gradient-descent","repo_url":"https://github.com/lixilinx/psgd_torch","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"preconditioned-stochastic-gradient-descent","repo_url":"https://github.com/lixilinx/psgd_tf","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":"2d-human-pose-estimation","task_name":"2D Human Pose Estimation"}],"methods":[{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1512.04202","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1512.04202"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lixilinx/psgd_tf","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lixilinx/psgd_torch","reach":null}],"summary":{"ran_fixture":1,"ran_honours":1,"ran_draft_wrong":3,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"8d6d14bd8c4c9bc3","entry":"init_kron","repo":"lixilinx/psgd_torch","repo_kind":"official","path":"simplest_standalone_psgd.py","file_url":"https://github.com/lixilinx/psgd_torch/blob/HEAD/simplest_standalone_psgd.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8d6d14bd8c4c9bc3"}},{"code_sha256_prefix":"eddcb98704cbe648","entry":"lift2single","repo":"lixilinx/psgd_torch","repo_kind":"official","path":"psgd.py","file_url":"https://github.com/lixilinx/psgd_torch/blob/HEAD/psgd.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eddcb98704cbe648"}},{"code_sha256_prefix":"e3c58792f0261fb7","entry":"norm_lower_bound_skh","repo":"lixilinx/psgd_torch","repo_kind":"official","path":"psgd.py","file_url":"https://github.com/lixilinx/psgd_torch/blob/HEAD/psgd.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e3c58792f0261fb7"}},{"code_sha256_prefix":"61416702d57451ae","entry":"norm_lower_bound_spd","repo":"lixilinx/psgd_torch","repo_kind":"official","path":"psgd.py","file_url":"https://github.com/lixilinx/psgd_torch/blob/HEAD/psgd.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"61416702d57451ae"}},{"code_sha256_prefix":"cc005e3400dac8f2","entry":"norm_lower_bound_spd","repo":"lixilinx/psgd_torch","repo_kind":"official","path":"simplest_standalone_psgd.py","file_url":"https://github.com/lixilinx/psgd_torch/blob/HEAD/simplest_standalone_psgd.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cc005e3400dac8f2"}},{"code_sha256_prefix":"17540df6389a595f","entry":"norm_lower_bound_skh","repo":"lixilinx/psgd_torch","repo_kind":"official","path":"simplest_standalone_psgd.py","file_url":"https://github.com/lixilinx/psgd_torch/blob/HEAD/simplest_standalone_psgd.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"17540df6389a595f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}