{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-computational-advantage-of-depth-learning","title":"The Computational Advantage of Depth: Learning High-Dimensional Hierarchical Functions with Gradient Descent","arxiv_id":"2502.13961","date":"2025-02-19","proceeding":null,"authors":["Yatin Dandi","Luca Pesce","Lenka Zdeborová","Florent Krzakala"],"abstract":"Understanding the advantages of deep neural networks trained by gradient descent (GD) compared to shallow models remains an open theoretical challenge. While the study of multi-index models with Gaussian data in high dimensions has provided analytical insights into the benefits of GD-trained neural networks over kernels, the role of depth in improving sample complexity and generalization in GD-trained networks remains poorly understood. In this paper, we introduce a class of target functions (single and multi-index Gaussian hierarchical targets) that incorporate a hierarchy of latent subspace dimensionalities. This framework enables us to analytically study the learning dynamics and generalization performance of deep networks compared to shallow ones in the high-dimensional limit. Specifically, our main theorem shows that feature learning with GD reduces the effective dimensionality, transforming a high-dimensional problem into a sequence of lower-dimensional ones. This enables learning the target function with drastically less samples than with shallow networks. While the results are proven in a controlled training setting, we also discuss more common training procedures and argue that they learn through the same mechanisms. These findings open the way to further quantitative studies of the crucial role of depth in learning hierarchical structures with deep networks.","url_abs":"https://arxiv.org/abs/2502.13961v1","url_pdf":"https://arxiv.org/pdf/2502.13961v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-computational-advantage-of-depth-learning","repo_url":"https://github.com/IdePHICS/ComputationalDepth","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.13961","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.13961"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/IdePHICS/ComputationalDepth","reach":null}],"summary":{"ran_violates":3,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"9fc83ab24e07ff31","entry":"He_2","repo":"IdePHICS/ComputationalDepth","repo_kind":"official","path":"Codes/utils.py","file_url":"https://github.com/IdePHICS/ComputationalDepth/blob/HEAD/Codes/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9fc83ab24e07ff31"}},{"code_sha256_prefix":"988623dc00a4cf9b","entry":"He_3","repo":"IdePHICS/ComputationalDepth","repo_kind":"official","path":"Codes/utils.py","file_url":"https://github.com/IdePHICS/ComputationalDepth/blob/HEAD/Codes/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"988623dc00a4cf9b"}},{"code_sha256_prefix":"2aaa4150b2a4a469","entry":"He_4","repo":"IdePHICS/ComputationalDepth","repo_kind":"official","path":"Codes/utils.py","file_url":"https://github.com/IdePHICS/ComputationalDepth/blob/HEAD/Codes/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2aaa4150b2a4a469"}},{"code_sha256_prefix":"85fe19709e3f23be","entry":"ComputeMetrics","repo":"IdePHICS/ComputationalDepth","repo_kind":"official","path":"Codes/RunExperiment.py","file_url":"https://github.com/IdePHICS/ComputationalDepth/blob/HEAD/Codes/RunExperiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"85fe19709e3f23be"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}