{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/large-scale-distributed-second-order","title":"Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks","arxiv_id":"1811.12019","date":"2018-11-29","proceeding":"CVPR 2019 6","authors":["Kazuki Osawa","Yohei Tsuji","Yuichiro Ueno","Akira Naruse","Rio Yokota","Satoshi Matsuoka"],"abstract":"Large-scale distributed training of deep neural networks suffer from the\ngeneralization gap caused by the increase in the effective mini-batch size.\nPrevious approaches try to solve this problem by varying the learning rate and\nbatch size over epochs and layers, or some ad hoc modification of the batch\nnormalization. We propose an alternative approach using a second-order\noptimization method that shows similar generalization capability to first-order\nmethods, but converges faster and can handle larger mini-batches. To test our\nmethod on a benchmark where highly optimized first-order methods are available\nas references, we train ResNet-50 on ImageNet. We converged to 75% Top-1\nvalidation accuracy in 35 epochs for mini-batch sizes under 16,384, and\nachieved 75% even with a mini-batch size of 131,072, which took only 978\niterations.","url_abs":"http://arxiv.org/abs/1811.12019v5","url_pdf":"http://arxiv.org/pdf/1811.12019v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"large-scale-distributed-second-order","repo_url":"https://github.com/gpauloski/kfac_pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"large-scale-distributed-second-order","repo_url":"https://github.com/lzhangbv/kfac_pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"large-scale-distributed-second-order","repo_url":"https://github.com/tyohei/chainerkfac","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1811.12019","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}