{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/initialization-and-regularization-of-1","title":"Initialization and Regularization of Factorized Neural Layers","arxiv_id":"2105.01029","date":"2021-05-03","proceeding":"ICLR 2021 1","authors":["Mikhail Khodak","Neil Tenenholtz","Lester Mackey","Nicolò Fusi"],"abstract":"Factorized layers--operations parameterized by products of two or more matrices--occur in a variety of deep learning contexts, including compressed model training, certain types of knowledge distillation, and multi-head self-attention architectures. We study how to initialize and regularize deep nets containing such layers, examining two simple, understudied schemes, spectral initialization and Frobenius decay, for improving their performance. The guiding insight is to design optimization routines for these networks that are as close as possible to that of their well-tuned, non-decomposed counterparts; we back this intuition with an analysis of how the initialization and regularization schemes impact training with gradient descent, drawing on modern attempts to understand the interplay of weight-decay and batch-normalization. Empirically, we highlight the benefits of spectral initialization and Frobenius decay across a variety of settings. In model compression, we show that they enable low-rank methods to significantly outperform both unstructured sparsity and tensor methods on the task of training low-memory residual networks; analogs of the schemes also improve the performance of tensor decomposition techniques. For knowledge distillation, Frobenius decay enables a simple, overcomplete baseline that yields a compact model from over-parameterized training without requiring retraining with or pruning a teacher network. Finally, we show how both schemes applied to multi-head attention lead to improved performance on both translation and unsupervised pre-training.","url_abs":"https://arxiv.org/abs/2105.01029v2","url_pdf":"https://arxiv.org/pdf/2105.01029v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"initialization-and-regularization-of-1","repo_url":"https://github.com/microsoft/fnl_paper","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"model-compression","task_name":"Model Compression"},{"task_slug":"tensor-decomposition","task_name":"Tensor Decomposition"},{"task_slug":"unsupervised-pre-training","task_name":"Unsupervised Pre-training"}],"methods":[{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"pruning","method_name":"Pruning"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2105.01029","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2105.01029"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/microsoft/fnl_paper","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/akamaster/pytorch_resnet_cifar10","reach":{"status":"ok","spdx":"BSD-2-Clause"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/NVIDIA/DeepLearningExamples","reach":null}],"summary":{"ran_honours":1,"ran_fixture":1,"unverified":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1},"found_in_text":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d3cf977044db121a","entry":"train","repo":"microsoft/fnl_paper","repo_kind":"official","path":"pytorch_resnet_cifar10/trainer.py","file_url":"https://github.com/microsoft/fnl_paper/blob/HEAD/pytorch_resnet_cifar10/trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d3cf977044db121a"}},{"code_sha256_prefix":"3043dac672377c96","entry":"validate","repo":"microsoft/fnl_paper","repo_kind":"official","path":"pytorch_resnet_cifar10/trainer.py","file_url":"https://github.com/microsoft/fnl_paper/blob/HEAD/pytorch_resnet_cifar10/trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3043dac672377c96"}},{"code_sha256_prefix":"f0c9a29156911331","entry":"accuracy","repo":"akamaster/pytorch_resnet_cifar10","repo_kind":"found_in_text","path":"trainer.py","file_url":"https://github.com/akamaster/pytorch_resnet_cifar10/blob/HEAD/trainer.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f0c9a29156911331"}},{"code_sha256_prefix":"abb8eb203bc21b75","entry":"validate","repo":"akamaster/pytorch_resnet_cifar10","repo_kind":"found_in_text","path":"trainer.py","file_url":"https://github.com/akamaster/pytorch_resnet_cifar10/blob/HEAD/trainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-2-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"abb8eb203bc21b75"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}