{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/training-neural-networks-without-gradients-a","title":"Training Neural Networks Without Gradients: A Scalable ADMM Approach","arxiv_id":"1605.02026","date":"2016-05-06","proceeding":null,"authors":["Gavin Taylor","Ryan Burmeister","Zheng Xu","Bharat Singh","Ankit Patel","Tom Goldstein"],"abstract":"With the growing importance of large network models and enormous training\ndatasets, GPUs have become increasingly necessary to train neural networks.\nThis is largely because conventional optimization algorithms rely on stochastic\ngradient methods that don't scale well to large numbers of cores in a cluster\nsetting. Furthermore, the convergence of all gradient methods, including batch\nmethods, suffers from common problems like saturation effects, poor\nconditioning, and saddle points. This paper explores an unconventional training\nmethod that uses alternating direction methods and Bregman iteration to train\nnetworks without gradient descent steps. The proposed method reduces the\nnetwork training problem to a sequence of minimization sub-steps that can each\nbe solved globally in closed form. The proposed method is advantageous because\nit avoids many of the caveats that make gradient methods slow on highly\nnon-convex problems. The method exhibits strong scaling in the distributed\nsetting, yielding linear speedups even when split over thousands of cores.","url_abs":"http://arxiv.org/abs/1605.02026v1","url_pdf":"http://arxiv.org/pdf/1605.02026v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"training-neural-networks-without-gradients-a","repo_url":"https://github.com/PotatoThanh/ADMM-NeuralNetworks","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"training-neural-networks-without-gradients-a","repo_url":"https://gitlab.umiacs.umd.edu/tomg/admm_nets","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1605.02026","atlas_url":"https://app.syntology.ai/?focus=1605.02026","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}