{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-unified-perspective-on-natural-gradient","title":"A Unified Perspective on Natural Gradient Variational Inference with Gaussian Mixture Models","arxiv_id":"2209.11533","date":"2022-09-23","proceeding":null,"authors":["Oleg Arenz","Philipp Dahlinger","Zihan Ye","Michael Volpp","Gerhard Neumann"],"abstract":"Variational inference with Gaussian mixture models (GMMs) enables learning of highly tractable yet multi-modal approximations of intractable target distributions with up to a few hundred dimensions. The two currently most effective methods for GMM-based variational inference, VIPS and iBayes-GMM, both employ independent natural gradient updates for the individual components and their weights. We show for the first time, that their derived updates are equivalent, although their practical implementations and theoretical guarantees differ. We identify several design choices that distinguish both approaches, namely with respect to sample selection, natural gradient estimation, stepsize adaptation, and whether trust regions are enforced or the number of components adapted. We argue that for both approaches, the quality of the learned approximations can heavily suffer from the respective design choices: By updating the individual components using samples from the mixture model, iBayes-GMM often fails to produce meaningful updates to low-weight components, and by using a zero-order method for estimating the natural gradient, VIPS scales badly to higher-dimensional problems. Furthermore, we show that information-geometric trust-regions (used by VIPS) are effective even when using first-order natural gradient estimates, and often outperform the improved Bayesian learning rule (iBLR) update used by iBayes-GMM. We systematically evaluate the effects of design choices and show that a hybrid approach significantly outperforms both prior works. Along with this work, we publish our highly modular and efficient implementation for natural gradient variational inference with Gaussian mixture models, which supports 432 different combinations of design choices, facilitates the reproduction of all our experiments, and may prove valuable for the practitioner.","url_abs":"https://arxiv.org/abs/2209.11533v2","url_pdf":"https://arxiv.org/pdf/2209.11533v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-unified-perspective-on-natural-gradient","repo_url":"https://github.com/olegarenz/gmmvi_reproducibility","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"variational-inference","task_name":"Variational Inference"}],"methods":[{"method_slug":"variational-inference","method_name":"Variational Inference"}],"datasets_introduced":[{"slug":"experimental-results-for-a-unified","name":"Experimental Results for \"A Unified Perspective on Natural Gradient Variational Inference with Gaussian Mixture Models\"","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2209.11533","atlas_url":"https://app.syntology.ai/?focus=2209.11533","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2209.11533"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/olegarenz/gmmvi_reproducibility","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9890f042f541fb81","entry":"get_sweep_ids","repo":"olegarenz/gmmvi_reproducibility","repo_kind":"official","path":"hyperopt/sweep_fetching.py","file_url":"https://github.com/olegarenz/gmmvi_reproducibility/blob/HEAD/hyperopt/sweep_fetching.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9890f042f541fb81"}},{"code_sha256_prefix":"35297fbf0ec02238","entry":"interpolate_curves","repo":"olegarenz/gmmvi_reproducibility","repo_kind":"official","path":"hyperopt/plot_sweeps.py","file_url":"https://github.com/olegarenz/gmmvi_reproducibility/blob/HEAD/hyperopt/plot_sweeps.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"35297fbf0ec02238"}},{"code_sha256_prefix":"3b376ad3e3cc439a","entry":"key_list_to_dict","repo":"olegarenz/gmmvi_reproducibility","repo_kind":"official","path":"hyperopt/wandb_sweep.py","file_url":"https://github.com/olegarenz/gmmvi_reproducibility/blob/HEAD/hyperopt/wandb_sweep.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3b376ad3e3cc439a"}},{"code_sha256_prefix":"7b49e925ad164ecf","entry":"plot_means","repo":"olegarenz/gmmvi_reproducibility","repo_kind":"official","path":"hyperopt/plot_sweeps.py","file_url":"https://github.com/olegarenz/gmmvi_reproducibility/blob/HEAD/hyperopt/plot_sweeps.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b49e925ad164ecf"}},{"code_sha256_prefix":"cee5b72242d5112c","entry":"plot_shaded","repo":"olegarenz/gmmvi_reproducibility","repo_kind":"official","path":"hyperopt/plot_sweeps.py","file_url":"https://github.com/olegarenz/gmmvi_reproducibility/blob/HEAD/hyperopt/plot_sweeps.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cee5b72242d5112c"}},{"code_sha256_prefix":"8a77e9269e12262b","entry":"summarize_sweep","repo":"olegarenz/gmmvi_reproducibility","repo_kind":"official","path":"hyperopt/sweep_fetching.py","file_url":"https://github.com/olegarenz/gmmvi_reproducibility/blob/HEAD/hyperopt/sweep_fetching.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a77e9269e12262b"}},{"code_sha256_prefix":"4683d3059e2bc1ef","entry":"get_best_elbos","repo":"olegarenz/gmmvi_reproducibility","repo_kind":"official","path":"hyperopt/table_exp1.py","file_url":"https://github.com/olegarenz/gmmvi_reproducibility/blob/HEAD/hyperopt/table_exp1.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4683d3059e2bc1ef"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}