{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/how-deep-neural-networks-learn-compositional","title":"How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model","arxiv_id":"2307.02129","date":"2023-07-05","proceeding":null,"authors":["Francesco Cagnetta","Leonardo Petrini","Umberto M. Tomasini","Alessandro Favero","Matthieu Wyart"],"abstract":"Deep learning algorithms demonstrate a surprising ability to learn high-dimensional tasks from limited examples. This is commonly attributed to the depth of neural networks, enabling them to build a hierarchy of abstract, low-dimensional data representations. However, how many training examples are required to learn such representations remains unknown. To quantitatively study this question, we introduce the Random Hierarchy Model: a family of synthetic tasks inspired by the hierarchical structure of language and images. The model is a classification task where each class corresponds to a group of high-level features, chosen among several equivalent groups associated with the same class. In turn, each feature corresponds to a group of sub-features chosen among several equivalent ones and so on, following a hierarchy of composition rules. We find that deep networks learn the task by developing internal representations invariant to exchanging equivalent groups. Moreover, the number of data required corresponds to the point where correlations between low-level features and classes become detectable. Overall, our results indicate how deep networks overcome the curse of dimensionality by building invariant representations, and provide an estimate of the number of data required to learn a hierarchical task.","url_abs":"https://arxiv.org/abs/2307.02129v5","url_pdf":"https://arxiv.org/pdf/2307.02129v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"how-deep-neural-networks-learn-compositional","repo_url":"https://github.com/pcsl-epfl/hierarchy-learning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[{"slug":"random-hierarchy-model","name":"Random Hierarchy Model","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2307.02129","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.02129"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pcsl-epfl/hierarchy-learning","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"unverified":7},"by_repo_kind":{"official":{"samples":11,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"37ea32a587a28954","entry":"bincount","repo":"pcsl-epfl/hierarchy-learning","repo_kind":"official","path":"measure_correlations.py","file_url":"https://github.com/pcsl-epfl/hierarchy-learning/blob/HEAD/measure_correlations.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"37ea32a587a28954"}},{"code_sha256_prefix":"afbf939ccf041c0c","entry":"global_unfold","repo":"pcsl-epfl/hierarchy-learning","repo_kind":"official","path":"models/gcnn.py","file_url":"https://github.com/pcsl-epfl/hierarchy-learning/blob/HEAD/models/gcnn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"afbf939ccf041c0c"}},{"code_sha256_prefix":"c23da6206919edb5","entry":"loss_func","repo":"pcsl-epfl/hierarchy-learning","repo_kind":"official","path":"optim_loss.py","file_url":"https://github.com/pcsl-epfl/hierarchy-learning/blob/HEAD/optim_loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c23da6206919edb5"}},{"code_sha256_prefix":"6aa1dc0f32c1012c","entry":"svc","repo":"pcsl-epfl/hierarchy-learning","repo_kind":"official","path":"main_kernel.py","file_url":"https://github.com/pcsl-epfl/hierarchy-learning/blob/HEAD/main_kernel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6aa1dc0f32c1012c"}},{"code_sha256_prefix":"601dcd53028064a8","entry":"kernel_regression","repo":"pcsl-epfl/hierarchy-learning","repo_kind":"official","path":"main_kernel.py","file_url":"https://github.com/pcsl-epfl/hierarchy-learning/blob/HEAD/main_kernel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"601dcd53028064a8"}},{"code_sha256_prefix":"b3dc3634c2468bbf","entry":"linear_svc","repo":"pcsl-epfl/hierarchy-learning","repo_kind":"official","path":"main_kernel.py","file_url":"https://github.com/pcsl-epfl/hierarchy-learning/blob/HEAD/main_kernel.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b3dc3634c2468bbf"}},{"code_sha256_prefix":"1679a6a10d313c08","entry":"measure_accuracy","repo":"pcsl-epfl/hierarchy-learning","repo_kind":"official","path":"optim_loss.py","file_url":"https://github.com/pcsl-epfl/hierarchy-learning/blob/HEAD/optim_loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1679a6a10d313c08"}},{"code_sha256_prefix":"2350fd228840f8ab","entry":"opt_algo","repo":"pcsl-epfl/hierarchy-learning","repo_kind":"official","path":"optim_loss.py","file_url":"https://github.com/pcsl-epfl/hierarchy-learning/blob/HEAD/optim_loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2350fd228840f8ab"}},{"code_sha256_prefix":"7abf45dc07ba9e5f","entry":"tensor_roll","repo":"pcsl-epfl/hierarchy-learning","repo_kind":"official","path":"models/gcnn.py","file_url":"https://github.com/pcsl-epfl/hierarchy-learning/blob/HEAD/models/gcnn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7abf45dc07ba9e5f"}},{"code_sha256_prefix":"20662162ea3dece1","entry":"two_layers","repo":"pcsl-epfl/hierarchy-learning","repo_kind":"official","path":"main_layerwise.py","file_url":"https://github.com/pcsl-epfl/hierarchy-learning/blob/HEAD/main_layerwise.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"20662162ea3dece1"}},{"code_sha256_prefix":"4ebc0e09cb353eac","entry":"unpickle","repo":"pcsl-epfl/hierarchy-learning","repo_kind":"official","path":"init.py","file_url":"https://github.com/pcsl-epfl/hierarchy-learning/blob/HEAD/init.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4ebc0e09cb353eac"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}