{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/natural-neural-networks","title":"Natural Neural Networks","arxiv_id":"1507.00210","date":"2015-07-01","proceeding":"NeurIPS 2015 12","authors":["Guillaume Desjardins","Karen Simonyan","Razvan Pascanu","Koray Kavukcuoglu"],"abstract":"We introduce Natural Neural Networks, a novel family of algorithms that speed\nup convergence by adapting their internal representation during training to\nimprove conditioning of the Fisher matrix. In particular, we show a specific\nexample that employs a simple and efficient reparametrization of the neural\nnetwork weights by implicitly whitening the representation obtained at each\nlayer, while preserving the feed-forward computation of the network. Such\nnetworks can be trained efficiently via the proposed Projected Natural Gradient\nDescent algorithm (PRONG), which amortizes the cost of these reparametrizations\nover many parameter updates and is closely related to the Mirror Descent online\nlearning algorithm. We highlight the benefits of our method on both\nunsupervised and supervised learning tasks, and showcase its scalability by\ntraining on the large-scale ImageNet Challenge dataset.","url_abs":"http://arxiv.org/abs/1507.00210v1","url_pdf":"http://arxiv.org/pdf/1507.00210v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"natural-neural-networks","repo_url":"https://github.com/awur978/Autoencoder","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1507.00210","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}