{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/variational-autoencoders-in-the-presence-of-1","title":"Variational autoencoders in the presence of low-dimensional data: landscape and implicit bias","arxiv_id":"2112.06868","date":"2021-12-13","proceeding":"ICLR 2022 4","authors":["Frederic Koehler","Viraj Mehta","Chenghui Zhou","Andrej Risteski"],"abstract":"Variational Autoencoders are one of the most commonly used generative models, particularly for image data. A prominent difficulty in training VAEs is data that is supported on a lower-dimensional manifold. Recent work by Dai and Wipf (2020) proposes a two-stage training algorithm for VAEs, based on a conjecture that in standard VAE training the generator will converge to a solution with 0 variance which is correctly supported on the ground truth manifold. They gave partial support for that conjecture by showing that some optima of the VAE loss do satisfy this property, but did not analyze the training dynamics. In this paper, we show that for linear encoders/decoders, the conjecture is true-that is the VAE training does recover a generator with support equal to the ground truth manifold-and does so due to an implicit bias of gradient descent rather than merely the VAE loss itself. In the nonlinear case, we show that VAE training frequently learns a higher-dimensional manifold which is a superset of the ground truth manifold.","url_abs":"https://arxiv.org/abs/2112.06868v2","url_pdf":"https://arxiv.org/pdf/2112.06868v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"variational-autoencoders-in-the-presence-of-1","repo_url":"https://github.com/virajmehta/vae-training","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2112.06868","atlas_url":"https://app.syntology.ai/?focus=2112.06868","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2112.06868"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/virajmehta/vae-training","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":4},"by_repo_kind":{"official":{"samples":5,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e925bc37f3763841","entry":"fill_diagonal","repo":"virajmehta/vae-training","repo_kind":"official","path":"networks.py","file_url":"https://github.com/virajmehta/vae-training/blob/HEAD/networks.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e925bc37f3763841"}},{"code_sha256_prefix":"325968886537f810","entry":"binary_cross_entropy","repo":"virajmehta/vae-training","repo_kind":"official","path":"networks.py","file_url":"https://github.com/virajmehta/vae-training/blob/HEAD/networks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"325968886537f810"}},{"code_sha256_prefix":"2267d7cb36bd8d35","entry":"leaky_relu","repo":"virajmehta/vae-training","repo_kind":"official","path":"utils.py","file_url":"https://github.com/virajmehta/vae-training/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2267d7cb36bd8d35"}},{"code_sha256_prefix":"2cc5c094d751d00a","entry":"relu","repo":"virajmehta/vae-training","repo_kind":"official","path":"utils.py","file_url":"https://github.com/virajmehta/vae-training/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2cc5c094d751d00a"}},{"code_sha256_prefix":"178cf26c5191a40c","entry":"sigmoid","repo":"virajmehta/vae-training","repo_kind":"official","path":"utils.py","file_url":"https://github.com/virajmehta/vae-training/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"178cf26c5191a40c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}