{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/understanding-optimization-in-deep-learning","title":"Understanding Optimization in Deep Learning with Central Flows","arxiv_id":"2410.24206","date":"2024-10-31","proceeding":null,"authors":["Jeremy M. Cohen","Alex Damian","Ameet Talwalkar","Zico Kolter","Jason D. Lee"],"abstract":"Optimization in deep learning remains poorly understood, even in the simple setting of deterministic (i.e. full-batch) training. A key difficulty is that much of an optimizer's behavior is implicitly determined by complex oscillatory dynamics, referred to as the \"edge of stability.\" The main contribution of this paper is to show that an optimizer's implicit behavior can be explicitly captured by a \"central flow:\" a differential equation which models the time-averaged optimization trajectory. We show that these flows can empirically predict long-term optimization trajectories of generic neural networks with a high degree of numerical accuracy. By interpreting these flows, we reveal for the first time 1) the precise sense in which RMSProp adapts to the local loss landscape, and 2) an \"acceleration via regularization\" mechanism, wherein adaptive optimizers implicitly navigate towards low-curvature regions in which they can take larger steps. This mechanism is key to the efficacy of these adaptive optimizers. Overall, we believe that central flows constitute a promising tool for reasoning about optimization in deep learning.","url_abs":"https://arxiv.org/abs/2410.24206v1","url_pdf":"https://arxiv.org/pdf/2410.24206v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"deep-learning","task_name":"Deep Learning"},{"task_slug":"navigate","task_name":"Navigate"}],"methods":[{"method_slug":"rmsprop","method_name":"RMSProp"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.24206","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.24206"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/centralflows/centralflows","reach":{"status":"ok"}}],"summary":{"unverified":7},"by_repo_kind":{"found_in_text":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"dfe0f3c3bd968909","entry":"align_evecs","repo":"centralflows/centralflows","repo_kind":"found_in_text","path":"src/eig_solvers.py","file_url":"https://github.com/centralflows/centralflows/blob/HEAD/src/eig_solvers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dfe0f3c3bd968909"}},{"code_sha256_prefix":"218b3424b0f106aa","entry":"create_safe_wrapper","repo":"centralflows/centralflows","repo_kind":"found_in_text","path":"src/lobpcg.py","file_url":"https://github.com/centralflows/centralflows/blob/HEAD/src/lobpcg.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"218b3424b0f106aa"}},{"code_sha256_prefix":"4a6a62054e847987","entry":"mse_binary_loss","repo":"centralflows/centralflows","repo_kind":"found_in_text","path":"src/loss_criterion.py","file_url":"https://github.com/centralflows/centralflows/blob/HEAD/src/loss_criterion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4a6a62054e847987"}},{"code_sha256_prefix":"6f78e479f3209da1","entry":"mse_categorical_loss","repo":"centralflows/centralflows","repo_kind":"found_in_text","path":"src/loss_criterion.py","file_url":"https://github.com/centralflows/centralflows/blob/HEAD/src/loss_criterion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6f78e479f3209da1"}},{"code_sha256_prefix":"0ef8c1200f922298","entry":"mse_loss","repo":"centralflows/centralflows","repo_kind":"found_in_text","path":"src/loss_criterion.py","file_url":"https://github.com/centralflows/centralflows/blob/HEAD/src/loss_criterion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0ef8c1200f922298"}},{"code_sha256_prefix":"a9d9b256ac561ca8","entry":"scipy_eig","repo":"centralflows/centralflows","repo_kind":"found_in_text","path":"src/eig_solvers.py","file_url":"https://github.com/centralflows/centralflows/blob/HEAD/src/eig_solvers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a9d9b256ac561ca8"}},{"code_sha256_prefix":"f642ad8b744082c7","entry":"torch_lobpcg","repo":"centralflows/centralflows","repo_kind":"found_in_text","path":"src/lobpcg.py","file_url":"https://github.com/centralflows/centralflows/blob/HEAD/src/lobpcg.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f642ad8b744082c7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}