{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/stochastic-collapse-how-gradient-noise-1","title":"Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworks","arxiv_id":"2306.04251","date":"2023-06-07","proceeding":"NeurIPS 2023 11","authors":["Feng Chen","Daniel Kunin","Atsushi Yamamura","Surya Ganguli"],"abstract":"In this work, we reveal a strong implicit bias of stochastic gradient descent (SGD) that drives overly expressive networks to much simpler subnetworks, thereby dramatically reducing the number of independent parameters, and improving generalization. To reveal this bias, we identify invariant sets, or subsets of parameter space that remain unmodified by SGD. We focus on two classes of invariant sets that correspond to simpler (sparse or low-rank) subnetworks and commonly appear in modern architectures. Our analysis uncovers that SGD exhibits a property of stochastic attractivity towards these simpler invariant sets. We establish a sufficient condition for stochastic attractivity based on a competition between the loss landscape's curvature around the invariant set and the noise introduced by stochastic gradients. Remarkably, we find that an increased level of noise strengthens attractivity, leading to the emergence of attractive invariant sets associated with saddle-points or local maxima of the train loss. We observe empirically the existence of attractive invariant sets in trained deep neural networks, implying that SGD dynamics often collapses to simple subnetworks with either vanishing or redundant neurons. We further demonstrate how this simplifying process of stochastic collapse benefits generalization in a linear teacher-student framework. Finally, through this analysis, we mechanistically explain why early training with large learning rates for extended periods benefits subsequent generalization.","url_abs":"https://arxiv.org/abs/2306.04251v3","url_pdf":"https://arxiv.org/pdf/2306.04251v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"stochastic-collapse-how-gradient-noise-1","repo_url":"https://github.com/ccffccffcc/stochastic_collapse","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.04251","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.04251"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ccffccffcc/stochastic_collapse","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"825ac88080cdc65d","entry":"forward","repo":"ccffccffcc/stochastic_collapse","repo_kind":"official","path":"run/run_teacher_student_diagonal.py","file_url":"https://github.com/ccffccffcc/stochastic_collapse/blob/HEAD/run/run_teacher_student_diagonal.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"825ac88080cdc65d"}},{"code_sha256_prefix":"dc9a611d35e90a18","entry":"forward","repo":"ccffccffcc/stochastic_collapse","repo_kind":"official","path":"run/run_teacher_student_general.py","file_url":"https://github.com/ccffccffcc/stochastic_collapse/blob/HEAD/run/run_teacher_student_general.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dc9a611d35e90a18"}},{"code_sha256_prefix":"ac76a89a41c4f6bf","entry":"generate","repo":"ccffccffcc/stochastic_collapse","repo_kind":"official","path":"run/run_teacher_student_general.py","file_url":"https://github.com/ccffccffcc/stochastic_collapse/blob/HEAD/run/run_teacher_student_general.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ac76a89a41c4f6bf"}},{"code_sha256_prefix":"14cbc696d53a0fcd","entry":"warning_on_one_line","repo":"ccffccffcc/stochastic_collapse","repo_kind":"official","path":"run/exp.py","file_url":"https://github.com/ccffccffcc/stochastic_collapse/blob/HEAD/run/exp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"14cbc696d53a0fcd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}