{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pathologies-of-factorised-gaussian-and-mc","title":"On the Expressiveness of Approximate Inference in Bayesian Neural Networks","arxiv_id":"1909.00719","date":"2019-09-02","proceeding":"NeurIPS 2020 12","authors":["Andrew Y. K. Foong","David R. Burt","Yingzhen Li","Richard E. Turner"],"abstract":"While Bayesian neural networks (BNNs) hold the promise of being flexible, well-calibrated statistical models, inference often requires approximations whose consequences are poorly understood. We study the quality of common variational methods in approximating the Bayesian predictive distribution. For single-hidden layer ReLU BNNs, we prove a fundamental limitation in function-space of two of the most commonly used distributions defined in weight-space: mean-field Gaussian and Monte Carlo dropout. We find there are simple cases where neither method can have substantially increased uncertainty in between well-separated regions of low uncertainty. We provide strong empirical evidence that exact inference does not have this pathology, hence it is due to the approximation and not the model. In contrast, for deep networks, we prove a universality result showing that there exist approximate posteriors in the above classes which provide flexible uncertainty estimates. However, we find empirically that pathologies of a similar form as in the single-hidden layer case can persist when performing variational inference in deeper networks. Our results motivate careful consideration of the implications of approximate inference methods in BNNs.","url_abs":"https://arxiv.org/abs/1909.00719v4","url_pdf":"https://arxiv.org/pdf/1909.00719v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pathologies-of-factorised-gaussian-and-mc","repo_url":"https://github.com/cambridge-mlg/expressiveness-approx-bnns","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"pathologies-of-factorised-gaussian-and-mc","repo_url":"https://github.com/Daniil-Selikhanovych/bnn-vi","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"active-learning","task_name":"Active Learning"},{"task_slug":"bayesian-inference","task_name":"Bayesian Inference"},{"task_slug":"bayesian-optimisation","task_name":"Bayesian Optimisation"},{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"variational-inference","task_name":"Variational Inference"}],"methods":[{"method_slug":"relu","method_name":"ReLU"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1909.00719","atlas_url":"https://app.syntology.ai/?focus=1909.00719","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1909.00719"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cambridge-mlg/expressiveness-approx-bnns","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Daniil-Selikhanovych/bnn-vi","reach":null}],"summary":{"ran_honours":1,"unverified":12},"by_repo_kind":{"official":{"samples":10,"ran":0,"repositories":1},"listed":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1a781d75562f51b6","entry":"init_loc","repo":"Daniil-Selikhanovych/bnn-vi","repo_kind":"listed","path":"bnn_vi/models.py","file_url":"https://github.com/Daniil-Selikhanovych/bnn-vi/blob/HEAD/bnn_vi/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1a781d75562f51b6"}},{"code_sha256_prefix":"50c06ecbb9724c67","entry":"approximator_batch_loss","repo":"cambridge-mlg/expressiveness-approx-bnns","repo_kind":"official","path":"inbetween/losses.py","file_url":"https://github.com/cambridge-mlg/expressiveness-approx-bnns/blob/HEAD/inbetween/losses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"50c06ecbb9724c67"}},{"code_sha256_prefix":"9bd9e4cf6721073c","entry":"build_model_parameters","repo":"cambridge-mlg/expressiveness-approx-bnns","repo_kind":"official","path":"inbetween/utils.py","file_url":"https://github.com/cambridge-mlg/expressiveness-approx-bnns/blob/HEAD/inbetween/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9bd9e4cf6721073c"}},{"code_sha256_prefix":"053fda5bf30c7c10","entry":"elbo","repo":"cambridge-mlg/expressiveness-approx-bnns","repo_kind":"official","path":"inbetween/losses.py","file_url":"https://github.com/cambridge-mlg/expressiveness-approx-bnns/blob/HEAD/inbetween/losses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"053fda5bf30c7c10"}},{"code_sha256_prefix":"d8f7046532977b35","entry":"elbo_batch_loss","repo":"cambridge-mlg/expressiveness-approx-bnns","repo_kind":"official","path":"inbetween/losses.py","file_url":"https://github.com/cambridge-mlg/expressiveness-approx-bnns/blob/HEAD/inbetween/losses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d8f7046532977b35"}},{"code_sha256_prefix":"f1c0bec0ed30305b","entry":"gen_grid_inputs","repo":"cambridge-mlg/expressiveness-approx-bnns","repo_kind":"official","path":"inbetween/utils_2d.py","file_url":"https://github.com/cambridge-mlg/expressiveness-approx-bnns/blob/HEAD/inbetween/utils_2d.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1c0bec0ed30305b"}},{"code_sha256_prefix":"f5a56366a5365593","entry":"get_pickle_path","repo":"cambridge-mlg/expressiveness-approx-bnns","repo_kind":"official","path":"experiments/2d_experiments/box_plots.py","file_url":"https://github.com/cambridge-mlg/expressiveness-approx-bnns/blob/HEAD/experiments/2d_experiments/box_plots.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f5a56366a5365593"}},{"code_sha256_prefix":"2e7ec68d8dc7078f","entry":"load_2d_pred","repo":"cambridge-mlg/expressiveness-approx-bnns","repo_kind":"official","path":"inbetween/utils_2d.py","file_url":"https://github.com/cambridge-mlg/expressiveness-approx-bnns/blob/HEAD/inbetween/utils_2d.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2e7ec68d8dc7078f"}},{"code_sha256_prefix":"2f98c4ade9371a12","entry":"load_X_y","repo":"cambridge-mlg/expressiveness-approx-bnns","repo_kind":"official","path":"inbetween/utils_2d.py","file_url":"https://github.com/cambridge-mlg/expressiveness-approx-bnns/blob/HEAD/inbetween/utils_2d.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f98c4ade9371a12"}},{"code_sha256_prefix":"d78d069ab29d9e5e","entry":"overconfidence_ratios","repo":"cambridge-mlg/expressiveness-approx-bnns","repo_kind":"official","path":"experiments/2d_experiments/box_plots.py","file_url":"https://github.com/cambridge-mlg/expressiveness-approx-bnns/blob/HEAD/experiments/2d_experiments/box_plots.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d78d069ab29d9e5e"}},{"code_sha256_prefix":"ce6700f173ec94cf","entry":"plot_1D","repo":"Daniil-Selikhanovych/bnn-vi","repo_kind":"listed","path":"bnn_vi/plotting.py","file_url":"https://github.com/Daniil-Selikhanovych/bnn-vi/blob/HEAD/bnn_vi/plotting.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ce6700f173ec94cf"}},{"code_sha256_prefix":"e3f6dd6073b832eb","entry":"plot_2D","repo":"Daniil-Selikhanovych/bnn-vi","repo_kind":"listed","path":"bnn_vi/plotting.py","file_url":"https://github.com/Daniil-Selikhanovych/bnn-vi/blob/HEAD/bnn_vi/plotting.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e3f6dd6073b832eb"}},{"code_sha256_prefix":"d9edc8b294e512ae","entry":"process_data","repo":"cambridge-mlg/expressiveness-approx-bnns","repo_kind":"official","path":"inbetween/utils.py","file_url":"https://github.com/cambridge-mlg/expressiveness-approx-bnns/blob/HEAD/inbetween/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d9edc8b294e512ae"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}