{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/training-simplification-and-model","title":"Training Simplification and Model Simplification for Deep Learning: A Minimal Effort Back Propagation Method","arxiv_id":"1711.06528","date":"2017-11-17","proceeding":null,"authors":["Xu Sun","Xuancheng Ren","Shuming Ma","Bingzhen Wei","Wei Li","Jingjing Xu","Houfeng Wang","Yi Zhang"],"abstract":"We propose a simple yet effective technique to simplify the training and the\nresulting model of neural networks. In back propagation, only a small subset of\nthe full gradient is computed to update the model parameters. The gradient\nvectors are sparsified in such a way that only the top-k elements (in terms of\nmagnitude) are kept. As a result, only k rows or columns (depending on the\nlayout) of the weight matrix are modified, leading to a linear reduction in the\ncomputational cost. Based on the sparsified gradients, we further simplify the\nmodel by eliminating the rows or columns that are seldom updated, which will\nreduce the computational cost both in the training and decoding, and\npotentially accelerate decoding in real-world applications. Surprisingly,\nexperimental results demonstrate that most of time we only need to update fewer\nthan 5% of the weights at each back propagation pass. More interestingly, the\naccuracy of the resulting models is actually improved rather than degraded, and\na detailed analysis is given. The model simplification results show that we\ncould adaptively simplify the model which could often be reduced by around 9x,\nwithout any loss on accuracy or even with improved accuracy. The codes,\nincluding the extension, are available at https://github.com/lancopku/meSimp","url_abs":"http://arxiv.org/abs/1711.06528v2","url_pdf":"http://arxiv.org/pdf/1711.06528v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"training-simplification-and-model","repo_url":"https://github.com/jklj077/meProp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"training-simplification-and-model","repo_url":"https://github.com/lancopku/meProp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"training-simplification-and-model","repo_url":"https://github.com/lancopku/meSimp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}