{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lagging-inference-networks-and-posterior","title":"Lagging Inference Networks and Posterior Collapse in Variational Autoencoders","arxiv_id":"1901.05534","date":"2019-01-16","proceeding":"ICLR 2019 5","authors":["Junxian He","Daniel Spokoyny","Graham Neubig","Taylor Berg-Kirkpatrick"],"abstract":"The variational autoencoder (VAE) is a popular combination of deep latent\nvariable model and accompanying variational learning technique. By using a\nneural inference network to approximate the model's posterior on latent\nvariables, VAEs efficiently parameterize a lower bound on marginal data\nlikelihood that can be optimized directly via gradient methods. In practice,\nhowever, VAE training often results in a degenerate local optimum known as\n\"posterior collapse\" where the model learns to ignore the latent variable and\nthe approximate posterior mimics the prior. In this paper, we investigate\nposterior collapse from the perspective of training dynamics. We find that\nduring the initial stages of training the inference network fails to\napproximate the model's true posterior, which is a moving target. As a result,\nthe model is encouraged to ignore the latent encoding and posterior collapse\noccurs. Based on this observation, we propose an extremely simple modification\nto VAE training to reduce inference lag: depending on the model's current\nmutual information between latent variable and observation, we aggressively\noptimize the inference network before performing each model update. Despite\nintroducing neither new model components nor significant complexity over basic\nVAE, our approach is able to avoid the problem of collapse that has plagued a\nlarge amount of previous work. Empirically, our approach outperforms strong\nautoregressive baselines on text and image benchmarks in terms of held-out\nlikelihood, and is competitive with more complex techniques for avoiding\ncollapse while being substantially faster.","url_abs":"http://arxiv.org/abs/1901.05534v2","url_pdf":"http://arxiv.org/pdf/1901.05534v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lagging-inference-networks-and-posterior","repo_url":"https://github.com/jxhe/vae-lagging-encoder","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"lagging-inference-networks-and-posterior","repo_url":"https://github.com/spokoyny/spokoyny.github.io","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/text-generation-on-yahoo-questions","task":"Text Generation","dataset":"Yahoo Questions","model":"Aggressive VAE","rank_in_archive_order":1,"of":3,"metrics":{"KL":"5.7","NLL":"326.7","Perplexity":"59.7"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/1901.05534","atlas_url":"https://app.syntology.ai/?focus=1901.05534","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1901.05534"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jxhe/vae-lagging-encoder","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/spokoyny/spokoyny.github.io","reach":{"status":"ok"}}],"summary":{"ran":1,"ran_draft_wrong":1,"unverified":8},"by_repo_kind":{"official":{"samples":10,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"69f9f574272f9c32","entry":"generate_grid","repo":"jxhe/vae-lagging-encoder","repo_kind":"official","path":"modules/utils.py","file_url":"https://github.com/jxhe/vae-lagging-encoder/blob/HEAD/modules/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"69f9f574272f9c32"}},{"code_sha256_prefix":"5efe6eb2209f6591","entry":"log_sum_exp","repo":"jxhe/vae-lagging-encoder","repo_kind":"official","path":"modules/utils.py","file_url":"https://github.com/jxhe/vae-lagging-encoder/blob/HEAD/modules/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5efe6eb2209f6591"}},{"code_sha256_prefix":"6fcc55204af86253","entry":"calc_au","repo":"jxhe/vae-lagging-encoder","repo_kind":"official","path":"image.py","file_url":"https://github.com/jxhe/vae-lagging-encoder/blob/HEAD/image.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6fcc55204af86253"}},{"code_sha256_prefix":"831c1b0d4145f843","entry":"calc_iwnll","repo":"jxhe/vae-lagging-encoder","repo_kind":"official","path":"text.py","file_url":"https://github.com/jxhe/vae-lagging-encoder/blob/HEAD/text.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"831c1b0d4145f843"}},{"code_sha256_prefix":"9cf81d81989d6812","entry":"calc_mi","repo":"jxhe/vae-lagging-encoder","repo_kind":"official","path":"image.py","file_url":"https://github.com/jxhe/vae-lagging-encoder/blob/HEAD/image.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9cf81d81989d6812"}},{"code_sha256_prefix":"2d2593fbf606c876","entry":"calc_mi","repo":"jxhe/vae-lagging-encoder","repo_kind":"official","path":"text.py","file_url":"https://github.com/jxhe/vae-lagging-encoder/blob/HEAD/text.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2d2593fbf606c876"}},{"code_sha256_prefix":"135b3dc835ffe6ad","entry":"get_confirm_token","repo":"jxhe/vae-lagging-encoder","repo_kind":"official","path":"prepare_data.py","file_url":"https://github.com/jxhe/vae-lagging-encoder/blob/HEAD/prepare_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"135b3dc835ffe6ad"}},{"code_sha256_prefix":"79e431252f32cf60","entry":"test","repo":"jxhe/vae-lagging-encoder","repo_kind":"official","path":"image.py","file_url":"https://github.com/jxhe/vae-lagging-encoder/blob/HEAD/image.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"79e431252f32cf60"}},{"code_sha256_prefix":"3884f3eb2fb3adc2","entry":"test","repo":"jxhe/vae-lagging-encoder","repo_kind":"official","path":"text.py","file_url":"https://github.com/jxhe/vae-lagging-encoder/blob/HEAD/text.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3884f3eb2fb3adc2"}},{"code_sha256_prefix":"3d3feb42f9c79a9e","entry":"test","repo":"jxhe/vae-lagging-encoder","repo_kind":"official","path":"toy.py","file_url":"https://github.com/jxhe/vae-lagging-encoder/blob/HEAD/toy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3d3feb42f9c79a9e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}