{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gradient-descent-on-neural-networks-typically-1","title":"Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability","arxiv_id":"2103.00065","date":"2021-02-26","proceeding":"ICLR 2021 1","authors":["Jeremy M. Cohen","Simran Kaur","Yuanzhi Li","J. Zico Kolter","Ameet Talwalkar"],"abstract":"We empirically demonstrate that full-batch gradient descent on neural network training objectives typically operates in a regime we call the Edge of Stability. In this regime, the maximum eigenvalue of the training loss Hessian hovers just above the numerical value $2 / \\text{(step size)}$, and the training loss behaves non-monotonically over short timescales, yet consistently decreases over long timescales. Since this behavior is inconsistent with several widespread presumptions in the field of optimization, our findings raise questions as to whether these presumptions are relevant to neural network training. We hope that our findings will inspire future efforts aimed at rigorously understanding optimization at the Edge of Stability. Code is available at https://github.com/locuslab/edge-of-stability.","url_abs":"https://arxiv.org/abs/2103.00065v3","url_pdf":"https://arxiv.org/pdf/2103.00065v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"gradient-descent-on-neural-networks-typically-1","repo_url":"https://github.com/locuslab/edge-of-stability","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2103.00065","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2103.00065"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/chengyangfu/pytorch-vgg-cifar10","reach":{"status":"unanswered"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/hongyi-zhang/Fixup","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/locuslab/edge-of-stability","reach":{"status":"ok"}}],"summary":{"unverified":2},"by_repo_kind":{"found_in_text":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"9b8289076669fe4f","entry":"accuracy","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"9b8289076669fe4f"}},{"code_sha256_prefix":"d07d18ed4137d374","entry":"validate","repo":"hongyi-zhang/Fixup","repo_kind":"found_in_text","path":"imagenet/imagenet_train.py","file_url":"https://github.com/hongyi-zhang/Fixup/blob/HEAD/imagenet/imagenet_train.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"d07d18ed4137d374"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}