{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/identifying-and-attacking-the-saddle-point","title":"Identifying and attacking the saddle point problem in high-dimensional non-convex optimization","arxiv_id":"1406.2572","date":"2014-06-10","proceeding":"NeurIPS 2014 12","authors":["Yann Dauphin","Razvan Pascanu","Caglar Gulcehre","Kyunghyun Cho","Surya Ganguli","Yoshua Bengio"],"abstract":"A central challenge to many fields of science and engineering involves\nminimizing non-convex error functions over continuous, high dimensional spaces.\nGradient descent or quasi-Newton methods are almost ubiquitously used to\nperform such minimizations, and it is often thought that a main source of\ndifficulty for these local methods to find the global minimum is the\nproliferation of local minima with much higher error than the global minimum.\nHere we argue, based on results from statistical physics, random matrix theory,\nneural network theory, and empirical evidence, that a deeper and more profound\ndifficulty originates from the proliferation of saddle points, not local\nminima, especially in high dimensional problems of practical interest. Such\nsaddle points are surrounded by high error plateaus that can dramatically slow\ndown learning, and give the illusory impression of the existence of a local\nminimum. Motivated by these arguments, we propose a new approach to\nsecond-order optimization, the saddle-free Newton method, that can rapidly\nescape high dimensional saddle points, unlike gradient descent and quasi-Newton\nmethods. We apply this algorithm to deep or recurrent neural network training,\nand provide numerical evidence for its superior optimization performance.","url_abs":"http://arxiv.org/abs/1406.2572v1","url_pdf":"http://arxiv.org/pdf/1406.2572v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"identifying-and-attacking-the-saddle-point","repo_url":"https://github.com/dave-fernandes/SaddleFreeOptimizer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"identifying-and-attacking-the-saddle-point","repo_url":"https://github.com/jchunn/Ambition","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"identifying-and-attacking-the-saddle-point","repo_url":"https://github.com/ltatzel/pytorchhessianfree","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"identifying-and-attacking-the-saddle-point","repo_url":"https://github.com/smdrozdov/saddle_free_newton","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1406.2572","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1406.2572"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/smdrozdov/saddle_free_newton","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dave-fernandes/SaddleFreeOptimizer","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ltatzel/pytorchhessianfree","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jchunn/Ambition","reach":{"status":"ok"}}],"summary":{"unverified":3},"by_repo_kind":{"listed":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5cef777435aec68b","entry":"dataset","repo":"dave-fernandes/SaddleFreeOptimizer","repo_kind":"listed","path":"mnist/dataset.py","file_url":"https://github.com/dave-fernandes/SaddleFreeOptimizer/blob/HEAD/mnist/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5cef777435aec68b"}},{"code_sha256_prefix":"0d65e022d1e2371e","entry":"download","repo":"dave-fernandes/SaddleFreeOptimizer","repo_kind":"listed","path":"mnist/dataset.py","file_url":"https://github.com/dave-fernandes/SaddleFreeOptimizer/blob/HEAD/mnist/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0d65e022d1e2371e"}},{"code_sha256_prefix":"aec170f547afa0ca","entry":"read32","repo":"dave-fernandes/SaddleFreeOptimizer","repo_kind":"listed","path":"mnist/dataset.py","file_url":"https://github.com/dave-fernandes/SaddleFreeOptimizer/blob/HEAD/mnist/dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aec170f547afa0ca"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}