{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/embrace-the-gap-vaes-perform-independent","title":"Embrace the Gap: VAEs Perform Independent Mechanism Analysis","arxiv_id":"2206.02416","date":"2022-06-06","proceeding":null,"authors":["Patrik Reizinger","Luigi Gresele","Jack Brady","Julius von Kügelgen","Dominik Zietlow","Bernhard Schölkopf","Georg Martius","Wieland Brendel","Michel Besserve"],"abstract":"Variational autoencoders (VAEs) are a popular framework for modeling complex data distributions; they can be efficiently trained via variational inference by maximizing the evidence lower bound (ELBO), at the expense of a gap to the exact (log-)marginal likelihood. While VAEs are commonly used for representation learning, it is unclear why ELBO maximization would yield useful representations, since unregularized maximum likelihood estimation cannot invert the data-generating process. Yet, VAEs often succeed at this task. We seek to elucidate this apparent paradox by studying nonlinear VAEs in the limit of near-deterministic decoders. We first prove that, in this regime, the optimal encoder approximately inverts the decoder -- a commonly used but unproven conjecture -- which we refer to as {\\em self-consistency}. Leveraging self-consistency, we show that the ELBO converges to a regularized log-likelihood. This allows VAEs to perform what has recently been termed independent mechanism analysis (IMA): it adds an inductive bias towards decoders with column-orthogonal Jacobians, which helps recovering the true latent factors. The gap between ELBO and log-likelihood is therefore welcome, since it bears unanticipated benefits for nonlinear representation learning. In experiments on synthetic and image data, we show that VAEs uncover the true latent factors when the data generating process satisfies the IMA assumption.","url_abs":"https://arxiv.org/abs/2206.02416v3","url_pdf":"https://arxiv.org/pdf/2206.02416v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"embrace-the-gap-vaes-perform-independent","repo_url":"https://github.com/rpatrik96/ima-vae","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"inductive-bias","task_name":"Inductive Bias"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"variational-inference","task_name":"Variational Inference"}],"methods":[{"method_slug":"variational-inference","method_name":"Variational Inference"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2206.02416","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.02416"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rpatrik96/ima-vae","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"unverified":14},"by_repo_kind":{"official":{"samples":15,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ff5945eb18d03970","entry":"make_cost_matrix","repo":"rpatrik96/ima-vae","repo_kind":"official","path":"ima_vae/metrics/mcc.py","file_url":"https://github.com/rpatrik96/ima-vae/blob/HEAD/ima_vae/metrics/mcc.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ff5945eb18d03970"}},{"code_sha256_prefix":"119ad9c491e1ed7f","entry":"amari_distance","repo":"rpatrik96/ima-vae","repo_kind":"official","path":"ima_vae/metrics/amari.py","file_url":"https://github.com/rpatrik96/ima-vae/blob/HEAD/ima_vae/metrics/amari.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"119ad9c491e1ed7f"}},{"code_sha256_prefix":"ebde3b734314d2fc","entry":"batch_jacobian","repo":"rpatrik96/ima-vae","repo_kind":"official","path":"ima_vae/utils.py","file_url":"https://github.com/rpatrik96/ima-vae/blob/HEAD/ima_vae/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ebde3b734314d2fc"}},{"code_sha256_prefix":"78c1fa7888a78cf5","entry":"cima_kl_diagonality","repo":"rpatrik96/ima-vae","repo_kind":"official","path":"ima_vae/metrics/cima.py","file_url":"https://github.com/rpatrik96/ima-vae/blob/HEAD/ima_vae/metrics/cima.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"78c1fa7888a78cf5"}},{"code_sha256_prefix":"f7a92cf3d1031256","entry":"col_norms","repo":"rpatrik96/ima-vae","repo_kind":"official","path":"ima_vae/metrics/conformal.py","file_url":"https://github.com/rpatrik96/ima-vae/blob/HEAD/ima_vae/metrics/conformal.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f7a92cf3d1031256"}},{"code_sha256_prefix":"048ffc98e750b58a","entry":"compute_mig_with_discrete_factors","repo":"rpatrik96/ima-vae","repo_kind":"official","path":"ima_vae/metrics/mig.py","file_url":"https://github.com/rpatrik96/ima-vae/blob/HEAD/ima_vae/metrics/mig.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"048ffc98e750b58a"}},{"code_sha256_prefix":"b7ad76ea349de2d9","entry":"conformal_contrast","repo":"rpatrik96/ima-vae","repo_kind":"official","path":"ima_vae/metrics/conformal.py","file_url":"https://github.com/rpatrik96/ima-vae/blob/HEAD/ima_vae/metrics/conformal.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b7ad76ea349de2d9"}},{"code_sha256_prefix":"0152ff71b8ba45fe","entry":"correlation","repo":"rpatrik96/ima-vae","repo_kind":"official","path":"ima_vae/metrics/mcc.py","file_url":"https://github.com/rpatrik96/ima-vae/blob/HEAD/ima_vae/metrics/mcc.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0152ff71b8ba45fe"}},{"code_sha256_prefix":"0335cbb982e4e9cc","entry":"frobenius_diagonality","repo":"rpatrik96/ima-vae","repo_kind":"official","path":"ima_vae/metrics/conformal.py","file_url":"https://github.com/rpatrik96/ima-vae/blob/HEAD/ima_vae/metrics/conformal.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0335cbb982e4e9cc"}},{"code_sha256_prefix":"acc74785b01233e3","entry":"get_interp_name","repo":"rpatrik96/ima-vae","repo_kind":"official","path":"ima_vae/utils.py","file_url":"https://github.com/rpatrik96/ima-vae/blob/HEAD/ima_vae/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"acc74785b01233e3"}},{"code_sha256_prefix":"5cae75030ada9571","entry":"get_prediction_score","repo":"rpatrik96/ima-vae","repo_kind":"official","path":"ima_vae/metrics/mig.py","file_url":"https://github.com/rpatrik96/ima-vae/blob/HEAD/ima_vae/metrics/mig.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5cae75030ada9571"}},{"code_sha256_prefix":"1f48aaf39a559c76","entry":"get_save_name","repo":"rpatrik96/ima-vae","repo_kind":"official","path":"ima_vae/utils.py","file_url":"https://github.com/rpatrik96/ima-vae/blob/HEAD/ima_vae/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1f48aaf39a559c76"}},{"code_sha256_prefix":"9672393a0bfcc23e","entry":"get_sprites_models","repo":"rpatrik96/ima-vae","repo_kind":"official","path":"ima_vae/models/nets.py","file_url":"https://github.com/rpatrik96/ima-vae/blob/HEAD/ima_vae/models/nets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9672393a0bfcc23e"}},{"code_sha256_prefix":"8c5b57d3b395a183","entry":"get_synth_models","repo":"rpatrik96/ima-vae","repo_kind":"official","path":"ima_vae/models/nets.py","file_url":"https://github.com/rpatrik96/ima-vae/blob/HEAD/ima_vae/models/nets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8c5b57d3b395a183"}},{"code_sha256_prefix":"3ffdfd80343c4173","entry":"mutual_information","repo":"rpatrik96/ima-vae","repo_kind":"official","path":"ima_vae/metrics/mig.py","file_url":"https://github.com/rpatrik96/ima-vae/blob/HEAD/ima_vae/metrics/mig.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3ffdfd80343c4173"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}