{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dynamics-of-stochastic-gradient-descent-for","title":"Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup","arxiv_id":"1906.08632","date":"2019-06-18","proceeding":"NeurIPS 2019 12","authors":["Sebastian Goldt","Madhu S. Advani","Andrew M. Saxe","Florent Krzakala","Lenka Zdeborová"],"abstract":"Deep neural networks achieve stellar generalisation even when they have enough parameters to easily fit all their training data. We study this phenomenon by analysing the dynamics and the performance of over-parameterised two-layer neural networks in the teacher-student setup, where one network, the student, is trained on data generated by another network, called the teacher. We show how the dynamics of stochastic gradient descent (SGD) is captured by a set of differential equations and prove that this description is asymptotically exact in the limit of large inputs. Using this framework, we calculate the final generalisation error of student networks that have more parameters than their teachers. We find that the final generalisation error of the student increases with network size when training only the first layer, but stays constant or even decreases with size when training both layers. We show that these different behaviours have their root in the different solutions SGD finds for different activation functions. Our results indicate that achieving good generalisation in neural networks goes beyond the properties of SGD alone and depends on the interplay of at least the algorithm, the model architecture, and the data set.","url_abs":"https://arxiv.org/abs/1906.08632v2","url_pdf":"https://arxiv.org/pdf/1906.08632v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dynamics-of-stochastic-gradient-descent-for","repo_url":"https://github.com/sgoldt/nn2pp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"dynamics-of-stochastic-gradient-descent-for","repo_url":"https://github.com/sgoldt/pyscm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"dynamics-of-stochastic-gradient-descent-for","repo_url":"https://github.com/cristianoBY/nn2pp-torch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1906.08632","atlas_url":"https://app.syntology.ai/?focus=1906.08632","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1906.08632"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cristianoBY/nn2pp-torch","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sgoldt/pyscm","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sgoldt/nn2pp","reach":{"status":"ok"}}],"summary":{"ran":2,"unverified":2},"by_repo_kind":{"listed":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"712c00a354115810","entry":"teacher_predict","repo":"cristianoBY/nn2pp-torch","repo_kind":"listed","path":"teacher_dataset.py","file_url":"https://github.com/cristianoBY/nn2pp-torch/blob/HEAD/teacher_dataset.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"712c00a354115810"}},{"code_sha256_prefix":"0a441ec277fdfd93","entry":"test","repo":"cristianoBY/nn2pp-torch","repo_kind":"listed","path":"teacher_student.py","file_url":"https://github.com/cristianoBY/nn2pp-torch/blob/HEAD/teacher_student.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a441ec277fdfd93"}},{"code_sha256_prefix":"c7e5c74fdeac0e26","entry":"get_Q","repo":"cristianoBY/nn2pp-torch","repo_kind":"listed","path":"evaluate.py","file_url":"https://github.com/cristianoBY/nn2pp-torch/blob/HEAD/evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c7e5c74fdeac0e26"}},{"code_sha256_prefix":"5a253b9056a8fd70","entry":"get_R","repo":"cristianoBY/nn2pp-torch","repo_kind":"listed","path":"evaluate.py","file_url":"https://github.com/cristianoBY/nn2pp-torch/blob/HEAD/evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a253b9056a8fd70"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}