{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/crafting-heavy-tails-in-weight-matrix","title":"Crafting Heavy-Tails in Weight Matrix Spectrum without Gradient Noise","arxiv_id":"2406.04657","date":"2024-06-07","proceeding":null,"authors":["Vignesh Kothapalli","Tianyu Pang","Shenyang Deng","Zongmin Liu","Yaoqing Yang"],"abstract":"Training strategies for modern deep neural networks (NNs) tend to induce a heavy-tailed (HT) empirical spectral density (ESD) in the layer weights. While previous efforts have shown that the HT phenomenon correlates with good generalization in large NNs, a theoretical explanation of its occurrence is still lacking. Especially, understanding the conditions which lead to this phenomenon can shed light on the interplay between generalization and weight spectra. Our work aims to bridge this gap by presenting a simple, rich setting to model the emergence of HT ESD. In particular, we present a theory-informed analysis for 'crafting' heavy tails in the ESD of two-layer NNs without any gradient noise. This is the first work to analyze a noise-free setting and incorporate optimizer (GD/Adam) dependent (large) learning rates into the HT ESD analysis. Our results highlight the role of learning rates on the Bulk+Spike and HT shape of the ESDs in the early phase of training, which can facilitate generalization in the two-layer NN. These observations shed light on the behavior of large-scale NNs, albeit in a much simpler setting. Last but not least, we present a novel perspective on the ESD evolution dynamics by analyzing the singular vectors of weight matrices and optimizer updates.","url_abs":"https://arxiv.org/abs/2406.04657v2","url_pdf":"https://arxiv.org/pdf/2406.04657v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"crafting-heavy-tails-in-weight-matrix","repo_url":"https://github.com/kvignesh1420/single-index-ht","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"adam","method_name":"Adam"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.04657","atlas_url":"https://app.syntology.ai/?focus=2406.04657","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.04657"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kvignesh1420/single-index-ht","reach":{"status":"ok"}}],"summary":{"ran":6},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"af651907e36ef135","entry":"get_student_model","repo":"kvignesh1420/single-index-ht","repo_kind":"official","path":"src/models.py","file_url":"https://github.com/kvignesh1420/single-index-ht/blob/HEAD/src/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"af651907e36ef135"}},{"code_sha256_prefix":"38801e12bc08eee3","entry":"get_teacher_model","repo":"kvignesh1420/single-index-ht","repo_kind":"official","path":"src/models.py","file_url":"https://github.com/kvignesh1420/single-index-ht/blob/HEAD/src/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"38801e12bc08eee3"}},{"code_sha256_prefix":"ccf42d5c20819210","entry":"prepare_data_hash","repo":"kvignesh1420/single-index-ht","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/kvignesh1420/single-index-ht/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ccf42d5c20819210"}},{"code_sha256_prefix":"cfd4a149f17e8111","entry":"prepare_dataloaders","repo":"kvignesh1420/single-index-ht","repo_kind":"official","path":"src/data.py","file_url":"https://github.com/kvignesh1420/single-index-ht/blob/HEAD/src/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cfd4a149f17e8111"}},{"code_sha256_prefix":"ce2e93b940fa25bb","entry":"prepare_student_model_hash","repo":"kvignesh1420/single-index-ht","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/kvignesh1420/single-index-ht/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ce2e93b940fa25bb"}},{"code_sha256_prefix":"6fa0051cd08bf372","entry":"prepare_teacher_model_hash","repo":"kvignesh1420/single-index-ht","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/kvignesh1420/single-index-ht/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6fa0051cd08bf372"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}