{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-shattered-gradients-problem-if-resnets","title":"The Shattered Gradients Problem: If resnets are the answer, then what is the question?","arxiv_id":"1702.08591","date":"2017-02-28","proceeding":"ICML 2017 8","authors":["David Balduzzi","Marcus Frean","Lennox Leary","JP Lewis","Kurt Wan-Duo Ma","Brian McWilliams"],"abstract":"A long-standing obstacle to progress in deep learning is the problem of\nvanishing and exploding gradients. Although, the problem has largely been\novercome via carefully constructed initializations and batch normalization,\narchitectures incorporating skip-connections such as highway and resnets\nperform much better than standard feedforward architectures despite well-chosen\ninitialization and batch normalization. In this paper, we identify the\nshattered gradients problem. Specifically, we show that the correlation between\ngradients in standard feedforward networks decays exponentially with depth\nresulting in gradients that resemble white noise whereas, in contrast, the\ngradients in architectures with skip-connections are far more resistant to\nshattering, decaying sublinearly. Detailed empirical evidence is presented in\nsupport of the analysis, on both fully-connected networks and convnets.\nFinally, we present a new \"looks linear\" (LL) initialization that prevents\nshattering, with preliminary experiments showing the new initialization allows\nto train very deep networks without the addition of skip-connections.","url_abs":"http://arxiv.org/abs/1702.08591v2","url_pdf":"http://arxiv.org/pdf/1702.08591v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-shattered-gradients-problem-if-resnets","repo_url":"https://github.com/LSLeary/LL-init","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1702.08591","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}