{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/stable-training-of-normalizing-flows-for-high","title":"Stable Training of Normalizing Flows for High-dimensional Variational Inference","arxiv_id":"2402.16408","date":"2024-02-26","proceeding":null,"authors":["Daniel Andrade"],"abstract":"Variational inference with normalizing flows (NFs) is an increasingly popular alternative to MCMC methods. In particular, NFs based on coupling layers (Real NVPs) are frequently used due to their good empirical performance. In theory, increasing the depth of normalizing flows should lead to more accurate posterior approximations. However, in practice, training deep normalizing flows for approximating high-dimensional posterior distributions is often infeasible due to the high variance of the stochastic gradients. In this work, we show that previous methods for stabilizing the variance of stochastic gradient descent can be insufficient to achieve stable training of Real NVPs. As the source of the problem, we identify that, during training, samples often exhibit unusual high values. As a remedy, we propose a combination of two methods: (1) soft-thresholding of the scale in Real NVPs, and (2) a bijective soft log transformation of the samples. We evaluate these and other previously proposed modification on several challenging target distributions, including a high-dimensional horseshoe logistic regression model. Our experiments show that with our modifications, stable training of Real NVPs for posteriors with several thousand dimensions is possible, allowing for more accurate marginal likelihood estimation via importance sampling. Moreover, we evaluate several common training techniques and architecture choices and provide practical advise for training NFs for high-dimensional variational inference.","url_abs":"https://arxiv.org/abs/2402.16408v1","url_pdf":"https://arxiv.org/pdf/2402.16408v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"stable-training-of-normalizing-flows-for-high","repo_url":"https://github.com/andrade-stats/normalizing-flows","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"variational-inference","task_name":"Variational Inference"}],"methods":[{"method_slug":"logistic-regression","method_name":"Logistic Regression"},{"method_slug":"normalizing-flows","method_name":"Normalizing Flows"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.16408","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.16408"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/andrade-stats/normalizing-flows","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":11,"unverified":4},"by_repo_kind":{"official":{"samples":15,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"4afcfcc45bcc2493","entry":"chi2_simple","repo":"andrade-stats/normalizing-flows","repo_kind":"official","path":"core_adjusted.py","file_url":"https://github.com/andrade-stats/normalizing-flows/blob/HEAD/core_adjusted.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4afcfcc45bcc2493"}},{"code_sha256_prefix":"822c2d0cbf925d06","entry":"getAllModuleIndices","repo":"andrade-stats/normalizing-flows","repo_kind":"official","path":"analysis.py","file_url":"https://github.com/andrade-stats/normalizing-flows/blob/HEAD/analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"822c2d0cbf925d06"}},{"code_sha256_prefix":"510987f49c20c772","entry":"getEmptyStatDic","repo":"andrade-stats/normalizing-flows","repo_kind":"official","path":"analysis.py","file_url":"https://github.com/andrade-stats/normalizing-flows/blob/HEAD/analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"510987f49c20c772"}},{"code_sha256_prefix":"399570b135a60e80","entry":"getNrFalseDetections","repo":"andrade-stats/normalizing-flows","repo_kind":"official","path":"evaluation.py","file_url":"https://github.com/andrade-stats/normalizing-flows/blob/HEAD/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"399570b135a60e80"}},{"code_sha256_prefix":"f8b7c4915f5875d0","entry":"getOutlierRecall","repo":"andrade-stats/normalizing-flows","repo_kind":"official","path":"evaluation.py","file_url":"https://github.com/andrade-stats/normalizing-flows/blob/HEAD/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f8b7c4915f5875d0"}},{"code_sha256_prefix":"8b2d5fe633436d0a","entry":"getVec","repo":"andrade-stats/normalizing-flows","repo_kind":"official","path":"analysis.py","file_url":"https://github.com/andrade-stats/normalizing-flows/blob/HEAD/analysis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8b2d5fe633436d0a"}},{"code_sha256_prefix":"94e580bb6e619979","entry":"get_log_prob_multivariate_t","repo":"andrade-stats/normalizing-flows","repo_kind":"official","path":"densities.py","file_url":"https://github.com/andrade-stats/normalizing-flows/blob/HEAD/densities.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"94e580bb6e619979"}},{"code_sha256_prefix":"40fd67c68c2dce9b","entry":"leakyClamp","repo":"andrade-stats/normalizing-flows","repo_kind":"official","path":"new_flows.py","file_url":"https://github.com/andrade-stats/normalizing-flows/blob/HEAD/new_flows.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"40fd67c68c2dce9b"}},{"code_sha256_prefix":"cbd75af666284269","entry":"showOutlierDetectionPerformance_power_fdr","repo":"andrade-stats/normalizing-flows","repo_kind":"official","path":"evaluation.py","file_url":"https://github.com/andrade-stats/normalizing-flows/blob/HEAD/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"cbd75af666284269"}},{"code_sha256_prefix":"add8722ad720659f","entry":"softClampAsymAdvanced","repo":"andrade-stats/normalizing-flows","repo_kind":"official","path":"new_flows.py","file_url":"https://github.com/andrade-stats/normalizing-flows/blob/HEAD/new_flows.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"add8722ad720659f"}},{"code_sha256_prefix":"c039b5cd7098af7d","entry":"softClampAsymNew","repo":"andrade-stats/normalizing-flows","repo_kind":"official","path":"new_flows.py","file_url":"https://github.com/andrade-stats/normalizing-flows/blob/HEAD/new_flows.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c039b5cd7098af7d"}},{"code_sha256_prefix":"70e480ffce9632e5","entry":"get_log_prob_inv_gamma","repo":"andrade-stats/normalizing-flows","repo_kind":"official","path":"densities.py","file_url":"https://github.com/andrade-stats/normalizing-flows/blob/HEAD/densities.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"70e480ffce9632e5"}},{"code_sha256_prefix":"8487032f2eeabb41","entry":"loadArray","repo":"andrade-stats/normalizing-flows","repo_kind":"official","path":"commons.py","file_url":"https://github.com/andrade-stats/normalizing-flows/blob/HEAD/commons.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8487032f2eeabb41"}},{"code_sha256_prefix":"8c4b93348da52ad9","entry":"loadStatistics","repo":"andrade-stats/normalizing-flows","repo_kind":"official","path":"commons.py","file_url":"https://github.com/andrade-stats/normalizing-flows/blob/HEAD/commons.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8c4b93348da52ad9"}},{"code_sha256_prefix":"50ad8e41c0b7de01","entry":"moveToDevice","repo":"andrade-stats/normalizing-flows","repo_kind":"official","path":"commons.py","file_url":"https://github.com/andrade-stats/normalizing-flows/blob/HEAD/commons.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"50ad8e41c0b7de01"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}