{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-a-continuous-time-model-of-gradient","title":"On a continuous time model of gradient descent dynamics and instability in deep learning","arxiv_id":"2302.01952","date":"2023-02-03","proceeding":null,"authors":["Mihaela Rosca","Yan Wu","Chongli Qin","Benoit Dherin"],"abstract":"The recipe behind the success of deep learning has been the combination of neural networks and gradient-based optimization. Understanding the behavior of gradient descent however, and particularly its instability, has lagged behind its empirical success. To add to the theoretical tools available to study gradient descent we propose the principal flow (PF), a continuous time flow that approximates gradient descent dynamics. To our knowledge, the PF is the only continuous flow that captures the divergent and oscillatory behaviors of gradient descent, including escaping local minima and saddle points. Through its dependence on the eigendecomposition of the Hessian the PF sheds light on the recently observed edge of stability phenomena in deep learning. Using our new understanding of instability we propose a learning rate adaptation method which enables us to control the trade-off between training stability and test set evaluation performance.","url_abs":"https://arxiv.org/abs/2302.01952v3","url_pdf":"https://arxiv.org/pdf/2302.01952v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-a-continuous-time-model-of-gradient","repo_url":"https://github.com/deepmind/discretisation_drift","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"on-a-continuous-time-model-of-gradient","repo_url":"https://github.com/deepmind/dd_two_player_games","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"deep-learning","task_name":"Deep Learning"}],"methods":[{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2302.01952","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2302.01952"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deepmind/discretisation_drift","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deepmind/dd_two_player_games","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":11},"by_repo_kind":{"listed":{"samples":11,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8ee04e47cc8ffe35","entry":"binary_cross_entropy_loss","repo":"deepmind/dd_two_player_games","repo_kind":"listed","path":"dd_two_player_games/losses.py","file_url":"https://github.com/deepmind/dd_two_player_games/blob/HEAD/dd_two_player_games/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8ee04e47cc8ffe35"}},{"code_sha256_prefix":"d217f26096c77c6d","entry":"calculate_frechet_distance_with_error_bars","repo":"deepmind/dd_two_player_games","repo_kind":"listed","path":"dd_two_player_games/metric_utils.py","file_url":"https://github.com/deepmind/dd_two_player_games/blob/HEAD/dd_two_player_games/metric_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d217f26096c77c6d"}},{"code_sha256_prefix":"91a2171efdef64bc","entry":"calculate_inception_score","repo":"deepmind/dd_two_player_games","repo_kind":"listed","path":"dd_two_player_games/metric_utils.py","file_url":"https://github.com/deepmind/dd_two_player_games/blob/HEAD/dd_two_player_games/metric_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"91a2171efdef64bc"}},{"code_sha256_prefix":"e1191eecfa7fae5f","entry":"calculate_inception_scores_with_error_bars","repo":"deepmind/dd_two_player_games","repo_kind":"listed","path":"dd_two_player_games/metric_utils.py","file_url":"https://github.com/deepmind/dd_two_player_games/blob/HEAD/dd_two_player_games/metric_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e1191eecfa7fae5f"}},{"code_sha256_prefix":"0c0abb6ad51c15e5","entry":"filter_out_new_training_state","repo":"deepmind/dd_two_player_games","repo_kind":"listed","path":"dd_two_player_games/gan.py","file_url":"https://github.com/deepmind/dd_two_player_games/blob/HEAD/dd_two_player_games/gan.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0c0abb6ad51c15e5"}},{"code_sha256_prefix":"0465af73b2877578","entry":"get_player_fn","repo":"deepmind/dd_two_player_games","repo_kind":"listed","path":"dd_two_player_games/gan.py","file_url":"https://github.com/deepmind/dd_two_player_games/blob/HEAD/dd_two_player_games/gan.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0465af73b2877578"}},{"code_sha256_prefix":"960e1119c4129674","entry":"image_postprocess","repo":"deepmind/dd_two_player_games","repo_kind":"listed","path":"dd_two_player_games/data_utils.py","file_url":"https://github.com/deepmind/dd_two_player_games/blob/HEAD/dd_two_player_games/data_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"960e1119c4129674"}},{"code_sha256_prefix":"801251546afebbdc","entry":"image_preprocess","repo":"deepmind/dd_two_player_games","repo_kind":"listed","path":"dd_two_player_games/data_utils.py","file_url":"https://github.com/deepmind/dd_two_player_games/blob/HEAD/dd_two_player_games/data_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"801251546afebbdc"}},{"code_sha256_prefix":"f870d8d7e9089108","entry":"sn_dict_for_logging","repo":"deepmind/dd_two_player_games","repo_kind":"listed","path":"dd_two_player_games/experiment.py","file_url":"https://github.com/deepmind/dd_two_player_games/blob/HEAD/dd_two_player_games/experiment.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f870d8d7e9089108"}},{"code_sha256_prefix":"f63e9d5b08ab1779","entry":"square_loss","repo":"deepmind/dd_two_player_games","repo_kind":"listed","path":"dd_two_player_games/losses.py","file_url":"https://github.com/deepmind/dd_two_player_games/blob/HEAD/dd_two_player_games/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f63e9d5b08ab1779"}},{"code_sha256_prefix":"daddd6a303235a22","entry":"square_loss_with_sigmoid_logits","repo":"deepmind/dd_two_player_games","repo_kind":"listed","path":"dd_two_player_games/losses.py","file_url":"https://github.com/deepmind/dd_two_player_games/blob/HEAD/dd_two_player_games/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"daddd6a303235a22"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}