{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fast-large-scale-optimization-by-unifying","title":"Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods","arxiv_id":"1311.2115","date":"2013-11-09","proceeding":null,"authors":["Jascha Sohl-Dickstein","Ben Poole","Surya Ganguli"],"abstract":"We present an algorithm for minimizing a sum of functions that combines the\ncomputational efficiency of stochastic gradient descent (SGD) with the second\norder curvature information leveraged by quasi-Newton methods. We unify these\ndisparate approaches by maintaining an independent Hessian approximation for\neach contributing function in the sum. We maintain computational tractability\nand limit memory requirements even for high dimensional optimization problems\nby storing and manipulating these quadratic approximations in a shared, time\nevolving, low dimensional subspace. Each update step requires only a single\ncontributing function or minibatch evaluation (as in SGD), and each step is\nscaled using an approximate inverse Hessian and little to no adjustment of\nhyperparameters is required (as is typical for quasi-Newton methods). This\nalgorithm contrasts with earlier stochastic second order techniques that treat\nthe Hessian of each contributing function as a noisy approximation to the full\nHessian, rather than as a target for direct estimation. We experimentally\ndemonstrate improved convergence on seven diverse optimization problems. The\nalgorithm is released as open source Python and MATLAB packages.","url_abs":"http://arxiv.org/abs/1311.2115v7","url_pdf":"http://arxiv.org/pdf/1311.2115v7.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fast-large-scale-optimization-by-unifying","repo_url":"https://github.com/Sohl-Dickstein/Sum-of-Functions-Optimizer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"computational-efficiency","task_name":"Computational Efficiency"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1311.2115","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}