{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tensor-normal-training-for-deep-learning","title":"Tensor Normal Training for Deep Learning Models","arxiv_id":"2106.02925","date":"2021-06-05","proceeding":"NeurIPS 2021 12","authors":["Yi Ren","Donald Goldfarb"],"abstract":"Despite the predominant use of first-order methods for training deep learning models, second-order methods, and in particular, natural gradient methods, remain of interest because of their potential for accelerating training through the use of curvature information. Several methods with non-diagonal preconditioning matrices, including KFAC, Shampoo, and K-BFGS, have been proposed and shown to be effective. Based on the so-called tensor normal (TN) distribution, we propose and analyze a brand new approximate natural gradient method, Tensor Normal Training (TNT), which like Shampoo, only requires knowledge of the shape of the training parameters. By approximating the probabilistically based Fisher matrix, as opposed to the empirical Fisher matrix, our method uses the block-wise covariance of the sampling based gradient as the pre-conditioning matrix. Moreover, the assumption that the sampling-based (tensor) gradient follows a TN distribution, ensures that its covariance has a Kronecker separable structure, which leads to a tractable approximation to the Fisher matrix. Consequently, TNT's memory requirements and per-iteration computational costs are only slightly higher than those for first-order methods. In our experiments, TNT exhibited superior optimization performance to state-of-the-art first-order methods, and comparable optimization performance to the state-of-the-art second-order methods KFAC and Shampoo. Moreover, TNT demonstrated its ability to generalize as well as first-order methods, while using fewer epochs.","url_abs":"https://arxiv.org/abs/2106.02925v2","url_pdf":"https://arxiv.org/pdf/2106.02925v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tensor-normal-training-for-deep-learning","repo_url":"https://github.com/renyiryry/tnt_neurips_2021","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"deep-learning","task_name":"Deep Learning"},{"task_slug":"second-order-methods","task_name":"Second-order methods"}],"methods":[{"method_slug":"distributed-shampoo","method_name":"Distributed Shampoo"},{"method_slug":"tnt","method_name":"TNT"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.02925","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.02925"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/renyiryry/tnt_neurips_2021","reach":null}],"summary":{"ran_draft_wrong":1,"ran_honours":1,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"c06b251e41179340","entry":"coupled_newton","repo":"renyiryry/tnt_neurips_2021","repo_kind":"official","path":"utils_git/utils.py","file_url":"https://github.com/renyiryry/tnt_neurips_2021/blob/HEAD/utils_git/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c06b251e41179340"}},{"code_sha256_prefix":"df029794796b1c0f","entry":"get_block_Fisher","repo":"renyiryry/tnt_neurips_2021","repo_kind":"official","path":"utils_git/utils.py","file_url":"https://github.com/renyiryry/tnt_neurips_2021/blob/HEAD/utils_git/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"df029794796b1c0f"}},{"code_sha256_prefix":"cb3278c590d00a7f","entry":"Fisher_BD_update","repo":"renyiryry/tnt_neurips_2021","repo_kind":"official","path":"utils_git/utils.py","file_url":"https://github.com/renyiryry/tnt_neurips_2021/blob/HEAD/utils_git/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cb3278c590d00a7f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}