{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/how-many-degrees-of-freedom-do-we-need-to","title":"How many degrees of freedom do we need to train deep networks: a loss landscape perspective","arxiv_id":"2107.05802","date":"2021-07-13","proceeding":"ICLR 2022 4","authors":["Brett W. Larsen","Stanislav Fort","Nic Becker","Surya Ganguli"],"abstract":"A variety of recent works, spanning pruning, lottery tickets, and training within random subspaces, have shown that deep neural networks can be trained using far fewer degrees of freedom than the total number of parameters. We analyze this phenomenon for random subspaces by first examining the success probability of hitting a training loss sub-level set when training within a random subspace of a given training dimensionality. We find a sharp phase transition in the success probability from $0$ to $1$ as the training dimension surpasses a threshold. This threshold training dimension increases as the desired final loss decreases, but decreases as the initial loss decreases. We then theoretically explain the origin of this phase transition, and its dependence on initialization and final desired loss, in terms of properties of the high-dimensional geometry of the loss landscape. In particular, we show via Gordon's escape theorem, that the training dimension plus the Gaussian width of the desired loss sub-level set, projected onto a unit sphere surrounding the initialization, must exceed the total number of parameters for the success probability to be large. In several architectures and datasets, we measure the threshold training dimension as a function of initialization and demonstrate that it is a small fraction of the total parameters, implying by our theory that successful training with so few dimensions is possible precisely because the Gaussian width of low loss sub-level sets is very large. Moreover, we compare this threshold training dimension to more sophisticated ways of reducing training degrees of freedom, including lottery tickets as well as a new, analogous method: lottery subspaces. Code is available at https://github.com/ganguli-lab/degrees-of-freedom.","url_abs":"https://arxiv.org/abs/2107.05802v2","url_pdf":"https://arxiv.org/pdf/2107.05802v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"how-many-degrees-of-freedom-do-we-need-to","repo_url":"https://github.com/ganguli-lab/degrees-of-freedom","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2107.05802","atlas_url":"https://app.syntology.ai/?focus=2107.05802","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2107.05802"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ganguli-lab/degrees-of-freedom","reach":null}],"summary":{"ran_draft_wrong":2,"unverified":4},"by_repo_kind":{"official":{"samples":6,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"82620ac387a3de49","entry":"cross_entropy_loss","repo":"ganguli-lab/degrees-of-freedom","repo_kind":"official","path":"lottery_subspace.py","file_url":"https://github.com/ganguli-lab/degrees-of-freedom/blob/HEAD/lottery_subspace.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"82620ac387a3de49"}},{"code_sha256_prefix":"d75f8e446fd2752b","entry":"flatten_leaves","repo":"ganguli-lab/degrees-of-freedom","repo_kind":"official","path":"training_utils.py","file_url":"https://github.com/ganguli-lab/degrees-of-freedom/blob/HEAD/training_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d75f8e446fd2752b"}},{"code_sha256_prefix":"4864ff72334718a3","entry":"generate_projection","repo":"ganguli-lab/degrees-of-freedom","repo_kind":"official","path":"training_utils.py","file_url":"https://github.com/ganguli-lab/degrees-of-freedom/blob/HEAD/training_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4864ff72334718a3"}},{"code_sha256_prefix":"b10110cfc5ddad5b","entry":"normal_accuracy","repo":"ganguli-lab/degrees-of-freedom","repo_kind":"official","path":"lottery_subspace.py","file_url":"https://github.com/ganguli-lab/degrees-of-freedom/blob/HEAD/lottery_subspace.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b10110cfc5ddad5b"}},{"code_sha256_prefix":"1d72592cc37b2066","entry":"normal_loss","repo":"ganguli-lab/degrees-of-freedom","repo_kind":"official","path":"lottery_subspace.py","file_url":"https://github.com/ganguli-lab/degrees-of-freedom/blob/HEAD/lottery_subspace.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1d72592cc37b2066"}},{"code_sha256_prefix":"bcc53283af041484","entry":"reform_leaves","repo":"ganguli-lab/degrees-of-freedom","repo_kind":"official","path":"training_utils.py","file_url":"https://github.com/ganguli-lab/degrees-of-freedom/blob/HEAD/training_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bcc53283af041484"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}