{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deep-linear-networks-for-regression-are","title":"Deep linear networks for regression are implicitly regularized towards flat minima","arxiv_id":"2405.13456","date":"2024-05-22","proceeding":null,"authors":["Pierre Marion","Lénaïc Chizat"],"abstract":"The largest eigenvalue of the Hessian, or sharpness, of neural networks is a key quantity to understand their optimization dynamics. In this paper, we study the sharpness of deep linear networks for univariate regression. Minimizers can have arbitrarily large sharpness, but not an arbitrarily small one. Indeed, we show a lower bound on the sharpness of minimizers, which grows linearly with depth. We then study the properties of the minimizer found by gradient flow, which is the limit of gradient descent with vanishing learning rate. We show an implicit regularization towards flat minima: the sharpness of the minimizer is no more than a constant times the lower bound. The constant depends on the condition number of the data covariance matrix, but not on width or depth. This result is proven both for a small-scale initialization and a residual initialization. Results of independent interest are shown in both cases. For small-scale initialization, we show that the learned weight matrices are approximately rank-one and that their singular vectors align. For residual initialization, convergence of the gradient flow for a Gaussian initialization of the residual network is proven. Numerical experiments illustrate our results and connect them to gradient descent with non-vanishing learning rate.","url_abs":"https://arxiv.org/abs/2405.13456v2","url_pdf":"https://arxiv.org/pdf/2405.13456v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deep-linear-networks-for-regression-are","repo_url":"https://github.com/pierremarion23/implicit-reg-sharpness","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"regression-1","task_name":"regression"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.13456","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.13456"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/PierreMarion23/implicit-reg-sharpness","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pierremarion23/implicit-reg-sharpness","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":8,"unverified":1},"by_repo_kind":{"official":{"samples":9,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5870180258b6c8a6","entry":"generate_data","repo":"PierreMarion23/implicit-reg-sharpness","repo_kind":"official","path":"utils.py","file_url":"https://github.com/PierreMarion23/implicit-reg-sharpness/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5870180258b6c8a6"}},{"code_sha256_prefix":"de709e50e91704f3","entry":"init_mlp","repo":"PierreMarion23/implicit-reg-sharpness","repo_kind":"official","path":"mlp.py","file_url":"https://github.com/PierreMarion23/implicit-reg-sharpness/blob/HEAD/mlp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"de709e50e91704f3"}},{"code_sha256_prefix":"a1f262ba71bc7b8f","entry":"init_resnet","repo":"PierreMarion23/implicit-reg-sharpness","repo_kind":"official","path":"resnet.py","file_url":"https://github.com/PierreMarion23/implicit-reg-sharpness/blob/HEAD/resnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a1f262ba71bc7b8f"}},{"code_sha256_prefix":"19ebd4e6ae48ea0d","entry":"linear_network","repo":"PierreMarion23/implicit-reg-sharpness","repo_kind":"official","path":"mlp.py","file_url":"https://github.com/PierreMarion23/implicit-reg-sharpness/blob/HEAD/mlp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"19ebd4e6ae48ea0d"}},{"code_sha256_prefix":"09c329e7eeefd796","entry":"linear_network_proj","repo":"PierreMarion23/implicit-reg-sharpness","repo_kind":"official","path":"resnet.py","file_url":"https://github.com/PierreMarion23/implicit-reg-sharpness/blob/HEAD/resnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"09c329e7eeefd796"}},{"code_sha256_prefix":"1225e1d2cfd73d73","entry":"loss_fn_resnet","repo":"PierreMarion23/implicit-reg-sharpness","repo_kind":"official","path":"resnet.py","file_url":"https://github.com/PierreMarion23/implicit-reg-sharpness/blob/HEAD/resnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1225e1d2cfd73d73"}},{"code_sha256_prefix":"a794995cb4ed8549","entry":"non_linear_network","repo":"PierreMarion23/implicit-reg-sharpness","repo_kind":"official","path":"mlp.py","file_url":"https://github.com/PierreMarion23/implicit-reg-sharpness/blob/HEAD/mlp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a794995cb4ed8549"}},{"code_sha256_prefix":"f19688ad8dd1a7e2","entry":"param_hessian_vector_product","repo":"PierreMarion23/implicit-reg-sharpness","repo_kind":"official","path":"utils.py","file_url":"https://github.com/PierreMarion23/implicit-reg-sharpness/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f19688ad8dd1a7e2"}},{"code_sha256_prefix":"f3f3e20037b2576c","entry":"hessian_vector_product","repo":"PierreMarion23/implicit-reg-sharpness","repo_kind":"official","path":"utils.py","file_url":"https://github.com/PierreMarion23/implicit-reg-sharpness/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f3f3e20037b2576c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}