{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/approaching-deep-learning-through-the","title":"Approaching Deep Learning through the Spectral Dynamics of Weights","arxiv_id":"2408.11804","date":"2024-08-21","proceeding":null,"authors":["David Yunis","Kumar Kshitij Patel","Samuel Wheeler","Pedro Savarese","Gal Vardi","Karen Livescu","Michael Maire","Matthew R. Walter"],"abstract":"We propose an empirical approach centered on the spectral dynamics of weights -- the behavior of singular values and vectors during optimization -- to unify and clarify several phenomena in deep learning. We identify a consistent bias in optimization across various experiments, from small-scale ``grokking'' to large-scale tasks like image classification with ConvNets, image generation with UNets, speech recognition with LSTMs, and language modeling with Transformers. We also demonstrate that weight decay enhances this bias beyond its role as a norm regularizer, even in practical systems. Moreover, we show that these spectral dynamics distinguish memorizing networks from generalizing ones, offering a novel perspective on this longstanding conundrum. Additionally, we leverage spectral dynamics to explore the emergence of well-performing sparse subnetworks (lottery tickets) and the structure of the loss surface through linear mode connectivity. Our findings suggest that spectral dynamics provide a coherent framework to better understand the behavior of neural networks across diverse settings.","url_abs":"https://arxiv.org/abs/2408.11804v1","url_pdf":"https://arxiv.org/pdf/2408.11804v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"approaching-deep-learning-through-the","repo_url":"https://github.com/dyunis/spectral_dynamics","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"deep-learning","task_name":"Deep Learning"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"linear-mode-connectivity","task_name":"Linear Mode Connectivity"},{"task_slug":"speech-recognition","task_name":"Speech Recognition"},{"task_slug":"image-classification","task_name":"image-classification"},{"task_slug":"speech-recognition-1","task_name":"speech-recognition"}],"methods":[{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2408.11804","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.11804"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dyunis/spectral_dynamics","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":9,"unverified":5},"by_repo_kind":{"official":{"samples":14,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"49bb1128d25e4dc4","entry":"conv2d_input_shapes","repo":"dyunis/spectral_dynamics","repo_kind":"official","path":"image_classification/model.py","file_url":"https://github.com/dyunis/spectral_dynamics/blob/HEAD/image_classification/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"49bb1128d25e4dc4"}},{"code_sha256_prefix":"1aad283571ee906a","entry":"evaluate","repo":"dyunis/spectral_dynamics","repo_kind":"official","path":"language_modeling/evaluate_run.py","file_url":"https://github.com/dyunis/spectral_dynamics/blob/HEAD/language_modeling/evaluate_run.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1aad283571ee906a"}},{"code_sha256_prefix":"c2e0451207d436cd","entry":"get_and_trim_cp_paths","repo":"dyunis/spectral_dynamics","repo_kind":"official","path":"language_modeling/make_branches_yaml.py","file_url":"https://github.com/dyunis/spectral_dynamics/blob/HEAD/language_modeling/make_branches_yaml.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c2e0451207d436cd"}},{"code_sha256_prefix":"d6f1b61e8dca7621","entry":"get_svd_dict","repo":"dyunis/spectral_dynamics","repo_kind":"official","path":"language_modeling/evaluate_lmc.py","file_url":"https://github.com/dyunis/spectral_dynamics/blob/HEAD/language_modeling/evaluate_lmc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d6f1b61e8dca7621"}},{"code_sha256_prefix":"f284942528713b0e","entry":"pl_ckpt_path_to_step","repo":"dyunis/spectral_dynamics","repo_kind":"official","path":"language_modeling/make_branches_yaml.py","file_url":"https://github.com/dyunis/spectral_dynamics/blob/HEAD/language_modeling/make_branches_yaml.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f284942528713b0e"}},{"code_sha256_prefix":"562e12a2970b7659","entry":"preprocess_datasets","repo":"dyunis/spectral_dynamics","repo_kind":"official","path":"language_modeling/dataset.py","file_url":"https://github.com/dyunis/spectral_dynamics/blob/HEAD/language_modeling/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"562e12a2970b7659"}},{"code_sha256_prefix":"3cf65ac55ede7d31","entry":"resnet_from_name","repo":"dyunis/spectral_dynamics","repo_kind":"official","path":"image_classification/model.py","file_url":"https://github.com/dyunis/spectral_dynamics/blob/HEAD/image_classification/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3cf65ac55ede7d31"}},{"code_sha256_prefix":"421c25a8e350f9cb","entry":"sv_prune_model_","repo":"dyunis/spectral_dynamics","repo_kind":"official","path":"language_modeling/evaluate_run.py","file_url":"https://github.com/dyunis/spectral_dynamics/blob/HEAD/language_modeling/evaluate_run.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"421c25a8e350f9cb"}},{"code_sha256_prefix":"5b1648aecd7770b5","entry":"valid_model_name","repo":"dyunis/spectral_dynamics","repo_kind":"official","path":"image_classification/model.py","file_url":"https://github.com/dyunis/spectral_dynamics/blob/HEAD/image_classification/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5b1648aecd7770b5"}},{"code_sha256_prefix":"73f3caf50cec419a","entry":"evaluate","repo":"dyunis/spectral_dynamics","repo_kind":"official","path":"language_modeling/evaluate_lmc.py","file_url":"https://github.com/dyunis/spectral_dynamics/blob/HEAD/language_modeling/evaluate_lmc.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"73f3caf50cec419a"}},{"code_sha256_prefix":"2751b3274de1defd","entry":"evaluate_lmc","repo":"dyunis/spectral_dynamics","repo_kind":"official","path":"language_modeling/evaluate_lmc.py","file_url":"https://github.com/dyunis/spectral_dynamics/blob/HEAD/language_modeling/evaluate_lmc.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2751b3274de1defd"}},{"code_sha256_prefix":"e9c261f899d592a9","entry":"evaluate_run","repo":"dyunis/spectral_dynamics","repo_kind":"official","path":"language_modeling/evaluate_run.py","file_url":"https://github.com/dyunis/spectral_dynamics/blob/HEAD/language_modeling/evaluate_run.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e9c261f899d592a9"}},{"code_sha256_prefix":"7c9887b6cb835a29","entry":"get_dataset","repo":"dyunis/spectral_dynamics","repo_kind":"official","path":"language_modeling/dataset.py","file_url":"https://github.com/dyunis/spectral_dynamics/blob/HEAD/language_modeling/dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7c9887b6cb835a29"}},{"code_sha256_prefix":"2c00218b3148eba4","entry":"setup_wandb","repo":"dyunis/spectral_dynamics","repo_kind":"official","path":"language_modeling/config.py","file_url":"https://github.com/dyunis/spectral_dynamics/blob/HEAD/language_modeling/config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2c00218b3148eba4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}