{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/residual-networks-behave-like-ensembles-of","title":"Residual Networks Behave Like Ensembles of Relatively Shallow Networks","arxiv_id":"1605.06431","date":"2016-05-20","proceeding":"NeurIPS 2016 12","authors":["Andreas Veit","Michael Wilber","Serge Belongie"],"abstract":"In this work we propose a novel interpretation of residual networks showing\nthat they can be seen as a collection of many paths of differing length.\nMoreover, residual networks seem to enable very deep networks by leveraging\nonly the short paths during training. To support this observation, we rewrite\nresidual networks as an explicit collection of paths. Unlike traditional\nmodels, paths through residual networks vary in length. Further, a lesion study\nreveals that these paths show ensemble-like behavior in the sense that they do\nnot strongly depend on each other. Finally, and most surprising, most paths are\nshorter than one might expect, and only the short paths are needed during\ntraining, as longer paths do not contribute any gradient. For example, most of\nthe gradient in a residual network with 110 layers comes from paths that are\nonly 10-34 layers deep. Our results reveal one of the key characteristics that\nseem to enable the training of very deep networks: Residual networks avoid the\nvanishing gradient problem by introducing short paths which can carry gradient\nthroughout the extent of very deep networks.","url_abs":"http://arxiv.org/abs/1605.06431v2","url_pdf":"http://arxiv.org/pdf/1605.06431v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"residual-networks-behave-like-ensembles-of","repo_url":"https://github.com/andreasveit/densenet-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"residual-networks-behave-like-ensembles-of","repo_url":"https://github.com/ml-research/rational_sl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1605.06431","atlas_url":"https://app.syntology.ai/?focus=1605.06431","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}