{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-batch-normalized-inference-network-keeps","title":"A Batch Normalized Inference Network Keeps the KL Vanishing Away","arxiv_id":"2004.12585","date":"2020-04-27","proceeding":"ACL 2020 6","authors":["Qile Zhu","Jianlin Su","Wei Bi","Xiaojiang Liu","Xiyao Ma","Xiaolin Li","Dapeng Wu"],"abstract":"Variational Autoencoder (VAE) is widely used as a generative model to approximate a model's posterior on latent variables by combining the amortized variational inference and deep neural networks. However, when paired with strong autoregressive decoders, VAE often converges to a degenerated local optimum known as \"posterior collapse\". Previous approaches consider the Kullback Leibler divergence (KL) individual for each datapoint. We propose to let the KL follow a distribution across the whole dataset, and analyze that it is sufficient to prevent posterior collapse by keeping the expectation of the KL's distribution positive. Then we propose Batch Normalized-VAE (BN-VAE), a simple but effective approach to set a lower bound of the expectation by regularizing the distribution of the approximate posterior's parameters. Without introducing any new model component or modifying the objective, our approach can avoid the posterior collapse effectively and efficiently. We further show that the proposed BN-VAE can be extended to conditional VAE (CVAE). Empirically, our approach surpasses strong autoregressive baselines on language modeling, text classification and dialogue generation, and rivals more complex approaches while keeping almost the same training time as VAE.","url_abs":"https://arxiv.org/abs/2004.12585v2","url_pdf":"https://arxiv.org/pdf/2004.12585v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-batch-normalized-inference-network-keeps","repo_url":"https://github.com/valdersoul/bn-vae","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"dialogue-generation","task_name":"Dialogue Generation"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"text-classification","task_name":"Text Classification"},{"task_slug":"variational-inference","task_name":"Variational Inference"},{"task_slug":"text-classification-1","task_name":"text-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2004.12585","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2004.12585"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/valdersoul/bn-vae","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"ran_honours":1,"unverified":9},"by_repo_kind":{"official":{"samples":11,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5efe6eb2209f6591","entry":"log_sum_exp","repo":"valdersoul/bn-vae","repo_kind":"official","path":"modules/utils.py","file_url":"https://github.com/valdersoul/bn-vae/blob/HEAD/modules/utils.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5efe6eb2209f6591"}},{"code_sha256_prefix":"6298a45a525420e8","entry":"safe_log","repo":"valdersoul/bn-vae","repo_kind":"official","path":"modules/utils.py","file_url":"https://github.com/valdersoul/bn-vae/blob/HEAD/modules/utils.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6298a45a525420e8"}},{"code_sha256_prefix":"ecb556d46a1e1a8e","entry":"calc_au","repo":"valdersoul/bn-vae","repo_kind":"official","path":"utils.py","file_url":"https://github.com/valdersoul/bn-vae/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ecb556d46a1e1a8e"}},{"code_sha256_prefix":"6df3d4cb4d4dc5c5","entry":"calc_iwnll","repo":"valdersoul/bn-vae","repo_kind":"official","path":"utils.py","file_url":"https://github.com/valdersoul/bn-vae/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6df3d4cb4d4dc5c5"}},{"code_sha256_prefix":"cd7e66b917a18914","entry":"calc_mi","repo":"valdersoul/bn-vae","repo_kind":"official","path":"utils.py","file_url":"https://github.com/valdersoul/bn-vae/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cd7e66b917a18914"}},{"code_sha256_prefix":"c8da81fe940d3ca2","entry":"create_exp_dir","repo":"valdersoul/bn-vae","repo_kind":"official","path":"exp_utils.py","file_url":"https://github.com/valdersoul/bn-vae/blob/HEAD/exp_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c8da81fe940d3ca2"}},{"code_sha256_prefix":"69f9f574272f9c32","entry":"generate_grid","repo":"valdersoul/bn-vae","repo_kind":"official","path":"modules/utils.py","file_url":"https://github.com/valdersoul/bn-vae/blob/HEAD/modules/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"69f9f574272f9c32"}},{"code_sha256_prefix":"135b3dc835ffe6ad","entry":"get_confirm_token","repo":"valdersoul/bn-vae","repo_kind":"official","path":"prepare_data.py","file_url":"https://github.com/valdersoul/bn-vae/blob/HEAD/prepare_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"135b3dc835ffe6ad"}},{"code_sha256_prefix":"2a751ef26f71f337","entry":"get_logger","repo":"valdersoul/bn-vae","repo_kind":"official","path":"exp_utils.py","file_url":"https://github.com/valdersoul/bn-vae/blob/HEAD/exp_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2a751ef26f71f337"}},{"code_sha256_prefix":"f1ba669623777faf","entry":"test","repo":"valdersoul/bn-vae","repo_kind":"official","path":"lm.py","file_url":"https://github.com/valdersoul/bn-vae/blob/HEAD/lm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1ba669623777faf"}},{"code_sha256_prefix":"7ad017041756a7ce","entry":"test","repo":"valdersoul/bn-vae","repo_kind":"official","path":"t_s_f.py","file_url":"https://github.com/valdersoul/bn-vae/blob/HEAD/t_s_f.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ad017041756a7ce"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}