{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-the-expressive-power-of-overlapping","title":"On the Expressive Power of Overlapping Architectures of Deep Learning","arxiv_id":"1703.02065","date":"2017-03-06","proceeding":"ICLR 2018 1","authors":["Or Sharir","Amnon Shashua"],"abstract":"Expressive efficiency refers to the relation between two architectures A and\nB, whereby any function realized by B could be replicated by A, but there\nexists functions realized by A, which cannot be replicated by B unless its size\ngrows significantly larger. For example, it is known that deep networks are\nexponentially efficient with respect to shallow networks, in the sense that a\nshallow network must grow exponentially large in order to approximate the\nfunctions represented by a deep network of polynomial size. In this work, we\nextend the study of expressive efficiency to the attribute of network\nconnectivity and in particular to the effect of \"overlaps\" in the convolutional\nprocess, i.e., when the stride of the convolution is smaller than its filter\nsize (receptive field). To theoretically analyze this aspect of network's\ndesign, we focus on a well-established surrogate for ConvNets called\nConvolutional Arithmetic Circuits (ConvACs), and then demonstrate empirically\nthat our results hold for standard ConvNets as well. Specifically, our analysis\nshows that having overlapping local receptive fields, and more broadly denser\nconnectivity, results in an exponential increase in the expressive capacity of\nneural networks. Moreover, while denser connectivity can increase the\nexpressive capacity, we show that the most common types of modern architectures\nalready exhibit exponential increase in expressivity, without relying on\nfully-connected layers.","url_abs":"http://arxiv.org/abs/1703.02065v4","url_pdf":"http://arxiv.org/pdf/1703.02065v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-the-expressive-power-of-overlapping","repo_url":"https://github.com/HUJI-Deep/OverlapsAndExpressiveness","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"deep-learning","task_name":"Deep Learning"}],"methods":[{"method_slug":"convolution","method_name":"Convolution"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1703.02065","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}