{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/traditional-and-heavy-tailed-self","title":"Traditional and Heavy-Tailed Self Regularization in Neural Network Models","arxiv_id":"1901.08276","date":"2019-01-24","proceeding":null,"authors":["Charles H. Martin","Michael W. Mahoney"],"abstract":"Random Matrix Theory (RMT) is applied to analyze the weight matrices of Deep\nNeural Networks (DNNs), including both production quality, pre-trained models\nsuch as AlexNet and Inception, and smaller models trained from scratch, such as\nLeNet5 and a miniature-AlexNet. Empirical and theoretical results clearly\nindicate that the empirical spectral density (ESD) of DNN layer matrices\ndisplays signatures of traditionally-regularized statistical models, even in\nthe absence of exogenously specifying traditional forms of regularization, such\nas Dropout or Weight Norm constraints. Building on recent results in RMT, most\nnotably its extension to Universality classes of Heavy-Tailed matrices, we\ndevelop a theory to identify \\emph{5+1 Phases of Training}, corresponding to\nincreasing amounts of \\emph{Implicit Self-Regularization}. For smaller and/or\nolder DNNs, this Implicit Self-Regularization is like traditional Tikhonov\nregularization, in that there is a `size scale' separating signal from noise.\nFor state-of-the-art DNNs, however, we identify a novel form of\n\\emph{Heavy-Tailed Self-Regularization}, similar to the self-organization seen\nin the statistical physics of disordered systems. This implicit\nSelf-Regularization can depend strongly on the many knobs of the training\nprocess. By exploiting the generalization gap phenomena, we demonstrate that we\ncan cause a small model to exhibit all 5+1 phases of training simply by\nchanging the batch size.","url_abs":"http://arxiv.org/abs/1901.08276v1","url_pdf":"http://arxiv.org/pdf/1901.08276v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"traditional-and-heavy-tailed-self","repo_url":"https://github.com/CalculatedContent/ImplicitSelfRegularization","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"traditional-and-heavy-tailed-self","repo_url":"https://github.com/CalculatedContent/WeightWatcher","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"grouped-convolution","method_name":"Grouped Convolution"},{"method_slug":"local-response-normalization","method_name":"Local Response Normalization"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1901.08276","atlas_url":"https://app.syntology.ai/?focus=1901.08276","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1901.08276"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/CalculatedContent/WeightWatcher","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/CalculatedContent/ImplicitSelfRegularization","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":4},"by_repo_kind":{"listed":{"samples":7,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0a2c8739a51e4984","entry":"apply_trap_mp_fit","repo":"CalculatedContent/WeightWatcher","repo_kind":"listed","path":"weightwatcher/remove_traps.py","file_url":"https://github.com/CalculatedContent/WeightWatcher/blob/HEAD/weightwatcher/remove_traps.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0a2c8739a51e4984"}},{"code_sha256_prefix":"792f4c10b5c1b21a","entry":"identify_trap_mode_indices","repo":"CalculatedContent/WeightWatcher","repo_kind":"listed","path":"weightwatcher/remove_traps.py","file_url":"https://github.com/CalculatedContent/WeightWatcher/blob/HEAD/weightwatcher/remove_traps.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"792f4c10b5c1b21a"}},{"code_sha256_prefix":"5fab4af8f547e332","entry":"summarize_trace","repo":"CalculatedContent/WeightWatcher","repo_kind":"listed","path":"weightwatcher/compute_trace.py","file_url":"https://github.com/CalculatedContent/WeightWatcher/blob/HEAD/weightwatcher/compute_trace.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5fab4af8f547e332"}},{"code_sha256_prefix":"64443cf24b9525f2","entry":"analyze_single_trap","repo":"CalculatedContent/WeightWatcher","repo_kind":"listed","path":"weightwatcher/remove_traps.py","file_url":"https://github.com/CalculatedContent/WeightWatcher/blob/HEAD/weightwatcher/remove_traps.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"64443cf24b9525f2"}},{"code_sha256_prefix":"b1fbf1fc91adc0a3","entry":"load_trap_bundle","repo":"CalculatedContent/WeightWatcher","repo_kind":"listed","path":"weightwatcher/trap_bundles.py","file_url":"https://github.com/CalculatedContent/WeightWatcher/blob/HEAD/weightwatcher/trap_bundles.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b1fbf1fc91adc0a3"}},{"code_sha256_prefix":"cb6ec82893bf637a","entry":"pl_compare","repo":"CalculatedContent/WeightWatcher","repo_kind":"listed","path":"weightwatcher/WW_powerlaw.py","file_url":"https://github.com/CalculatedContent/WeightWatcher/blob/HEAD/weightwatcher/WW_powerlaw.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cb6ec82893bf637a"}},{"code_sha256_prefix":"60345cbbb29b8e97","entry":"save_trap_bundle","repo":"CalculatedContent/WeightWatcher","repo_kind":"listed","path":"weightwatcher/trap_bundles.py","file_url":"https://github.com/CalculatedContent/WeightWatcher/blob/HEAD/weightwatcher/trap_bundles.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"60345cbbb29b8e97"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}