{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/luck-matters-understanding-training-dynamics","title":"Luck Matters: Understanding Training Dynamics of Deep ReLU Networks","arxiv_id":"1905.13405","date":"2019-05-31","proceeding":null,"authors":["Yuandong Tian","Tina Jiang","Qucheng Gong","Ari Morcos"],"abstract":"We analyze the dynamics of training deep ReLU networks and their implications on generalization capability. Using a teacher-student setting, we discovered a novel relationship between the gradient received by hidden student nodes and the activations of teacher nodes for deep ReLU networks. With this relationship and the assumption of small overlapping teacher node activations, we prove that (1) student nodes whose weights are initialized to be close to teacher nodes converge to them at a faster rate, and (2) in over-parameterized regimes and 2-layer case, while a small set of lucky nodes do converge to the teacher nodes, the fan-out weights of other nodes converge to zero. This framework provides insight into multiple puzzling phenomena in deep learning like over-parameterization, implicit regularization, lottery tickets, etc. We verify our assumption by showing that the majority of BatchNorm biases of pre-trained VGG11/16 models are negative. Experiments on (1) random deep teacher networks with Gaussian inputs, (2) teacher network pre-trained on CIFAR-10 and (3) extensive ablation studies validate our multiple theoretical predictions.","url_abs":"https://arxiv.org/abs/1905.13405v4","url_pdf":"https://arxiv.org/pdf/1905.13405v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"luck-matters-understanding-training-dynamics","repo_url":"https://github.com/facebookresearch/luckmatters","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"relu","method_name":"ReLU"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1905.13405","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1905.13405"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/luckmatters","reach":null}],"summary":{"ran_honours":2,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"4c4dcc0e2932ffd3","entry":"get_aug_w","repo":"facebookresearch/luckmatters","repo_kind":"official","path":"luckmatter/model_gen.py","file_url":"https://github.com/facebookresearch/luckmatters/blob/HEAD/luckmatter/model_gen.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4c4dcc0e2932ffd3"}},{"code_sha256_prefix":"61e4bd1ea55c4f67","entry":"haar_measure","repo":"facebookresearch/luckmatters","repo_kind":"official","path":"luckmatter/theory_utils.py","file_url":"https://github.com/facebookresearch/luckmatters/blob/HEAD/luckmatter/theory_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"61e4bd1ea55c4f67"}},{"code_sha256_prefix":"e4bfbd05fe8f8e2e","entry":"init_separate_w","repo":"facebookresearch/luckmatters","repo_kind":"official","path":"luckmatter/theory_utils.py","file_url":"https://github.com/facebookresearch/luckmatters/blob/HEAD/luckmatter/theory_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e4bfbd05fe8f8e2e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}