{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/block-normalized-gradient-method-an-empirical","title":"Block-Normalized Gradient Method: An Empirical Study for Training Deep Neural Network","arxiv_id":"1707.04822","date":"2017-07-16","proceeding":"ICLR 2018 1","authors":["Adams Wei Yu","Lei Huang","Qihang Lin","Ruslan Salakhutdinov","Jaime Carbonell"],"abstract":"In this paper, we propose a generic and simple strategy for utilizing\nstochastic gradient information in optimization. The technique essentially\ncontains two consecutive steps in each iteration: 1) computing and normalizing\neach block (layer) of the mini-batch stochastic gradient; 2) selecting\nappropriate step size to update the decision variable (parameter) towards the\nnegative of the block-normalized gradient. We conduct extensive empirical\nstudies on various non-convex neural network optimization problems, including\nmulti-layer perceptron, convolution neural networks and recurrent neural\nnetworks. The results indicate the block-normalized gradient can help\naccelerate the training of neural networks. In particular, we observe that the\nnormalized gradient methods having constant step size with occasionally decay,\nsuch as SGD with momentum, have better performance in the deep convolution\nneural networks, while those with adaptive step sizes, such as Adam, perform\nbetter in recurrent neural networks. Besides, we also observe this line of\nmethods can lead to solutions with better generalization properties, which is\nconfirmed by the performance improvement over strong baselines.","url_abs":"http://arxiv.org/abs/1707.04822v2","url_pdf":"http://arxiv.org/pdf/1707.04822v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"block-normalized-gradient-method-an-empirical","repo_url":"https://github.com/AliOsm/shakkelha","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"unanswered"}},{"paper_slug":"block-normalized-gradient-method-an-empirical","repo_url":"https://github.com/clcarwin/sphereface","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"unanswered"}}],"tasks":[],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1707.04822","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}