{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-gaussian-equivalence-of-generative-models","title":"The Gaussian equivalence of generative models for learning with shallow neural networks","arxiv_id":"2006.14709","date":"2020-06-25","proceeding":null,"authors":["Sebastian Goldt","Bruno Loureiro","Galen Reeves","Florent Krzakala","Marc Mézard","Lenka Zdeborová"],"abstract":"Understanding the impact of data structure on the computational tractability of learning is a key challenge for the theory of neural networks. Many theoretical works do not explicitly model training data, or assume that inputs are drawn component-wise independently from some simple probability distribution. Here, we go beyond this simple paradigm by studying the performance of neural networks trained on data drawn from pre-trained generative models. This is possible due to a Gaussian equivalence stating that the key metrics of interest, such as the training and test errors, can be fully captured by an appropriately chosen Gaussian model. We provide three strands of rigorous, analytical and numerical evidence corroborating this equivalence. First, we establish rigorous conditions for the Gaussian equivalence to hold in the case of single-layer generative models, as well as deterministic rates for convergence in distribution. Second, we leverage this equivalence to derive a closed set of equations describing the generalisation performance of two widely studied machine learning problems: two-layer neural networks trained using one-pass stochastic gradient descent, and full-batch pre-learned features or kernel methods. Finally, we perform experiments demonstrating how our theory applies to deep, pre-trained generative models. These results open a viable path to the theoretical study of machine learning models with realistic data.","url_abs":"https://arxiv.org/abs/2006.14709v3","url_pdf":"https://arxiv.org/pdf/2006.14709v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-gaussian-equivalence-of-generative-models","repo_url":"https://github.com/sgoldt/gaussian-equiv-2layer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2006.14709","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.14709"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sgoldt/gaussian-equiv-2layer","reach":null}],"summary":{"ran_honours":1,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"609bbb9ffe542360","entry":"get_eg_analytical","repo":"sgoldt/gaussian-equiv-2layer","repo_kind":"official","path":"deepgen_online.py","file_url":"https://github.com/sgoldt/gaussian-equiv-2layer/blob/HEAD/deepgen_online.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"609bbb9ffe542360"}},{"code_sha256_prefix":"90cde0ebc61a7e8d","entry":"get_samples","repo":"sgoldt/gaussian-equiv-2layer","repo_kind":"official","path":"deepgen_online.py","file_url":"https://github.com/sgoldt/gaussian-equiv-2layer/blob/HEAD/deepgen_online.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"90cde0ebc61a7e8d"}},{"code_sha256_prefix":"e032115652d9b6cf","entry":"eval_student","repo":"sgoldt/gaussian-equiv-2layer","repo_kind":"official","path":"deepgen_online.py","file_url":"https://github.com/sgoldt/gaussian-equiv-2layer/blob/HEAD/deepgen_online.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e032115652d9b6cf"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}