{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/being-bayesian-even-just-a-bit-fixes","title":"Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks","arxiv_id":"2002.10118","date":"2020-02-24","proceeding":"ICML 2020 1","authors":["Agustinus Kristiadi","Matthias Hein","Philipp Hennig"],"abstract":"The point estimates of ReLU classification networks---arguably the most widely used neural network architecture---have been shown to yield arbitrarily high confidence far away from the training data. This architecture, in conjunction with a maximum a posteriori estimation scheme, is thus not calibrated nor robust. Approximate Bayesian inference has been empirically demonstrated to improve predictive uncertainty in neural networks, although the theoretical analysis of such Bayesian approximations is limited. We theoretically analyze approximate Gaussian distributions on the weights of ReLU networks and show that they fix the overconfidence problem. Furthermore, we show that even a simplistic, thus cheap, Bayesian approximation, also fixes these issues. This indicates that a sufficient condition for a calibrated uncertainty on a ReLU network is \"to be a bit Bayesian\". These theoretical results validate the usage of last-layer Bayesian approximation and motivate a range of a fidelity-cost trade-off. We further validate these findings empirically via various standard experiments using common deep ReLU networks and Laplace approximations.","url_abs":"https://arxiv.org/abs/2002.10118v2","url_pdf":"https://arxiv.org/pdf/2002.10118v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"being-bayesian-even-just-a-bit-fixes","repo_url":"https://github.com/wiseodd/last_layer_laplace","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"bayesian-inference","task_name":"Bayesian Inference"}],"methods":[{"method_slug":"relu","method_name":"ReLU"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2002.10118","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2002.10118"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wiseodd/last_layer_laplace","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"unverified":8},"by_repo_kind":{"official":{"samples":8,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1caed47ba8deae4f","entry":"estimate_variance","repo":"wiseodd/last_layer_laplace","repo_kind":"official","path":"paper/laplace/llla.py","file_url":"https://github.com/wiseodd/last_layer_laplace/blob/HEAD/paper/laplace/llla.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"1caed47ba8deae4f"}},{"code_sha256_prefix":"5443601dbad4011c","entry":"estimate_variance","repo":"wiseodd/last_layer_laplace","repo_kind":"official","path":"paper/laplace/llla_binary.py","file_url":"https://github.com/wiseodd/last_layer_laplace/blob/HEAD/paper/laplace/llla_binary.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"5443601dbad4011c"}},{"code_sha256_prefix":"45fe82066cc3473a","entry":"exact_hessian","repo":"wiseodd/last_layer_laplace","repo_kind":"official","path":"hessian.py","file_url":"https://github.com/wiseodd/last_layer_laplace/blob/HEAD/hessian.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"45fe82066cc3473a"}},{"code_sha256_prefix":"5394a11ff85532c6","entry":"exact_hessian_diagonal_blocks","repo":"wiseodd/last_layer_laplace","repo_kind":"official","path":"hessian.py","file_url":"https://github.com/wiseodd/last_layer_laplace/blob/HEAD/hessian.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"5394a11ff85532c6"}},{"code_sha256_prefix":"9a87d1611a5e5414","entry":"gridsearch_var0","repo":"wiseodd/last_layer_laplace","repo_kind":"official","path":"paper/laplace/llla.py","file_url":"https://github.com/wiseodd/last_layer_laplace/blob/HEAD/paper/laplace/llla.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"9a87d1611a5e5414"}},{"code_sha256_prefix":"e8a7b6aaaa1ca3ce","entry":"gridsearch_var0","repo":"wiseodd/last_layer_laplace","repo_kind":"official","path":"paper/laplace/llla_binary.py","file_url":"https://github.com/wiseodd/last_layer_laplace/blob/HEAD/paper/laplace/llla_binary.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"e8a7b6aaaa1ca3ce"}},{"code_sha256_prefix":"00e88cac1828a093","entry":"predict","repo":"wiseodd/last_layer_laplace","repo_kind":"official","path":"paper/laplace/util.py","file_url":"https://github.com/wiseodd/last_layer_laplace/blob/HEAD/paper/laplace/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"00e88cac1828a093"}},{"code_sha256_prefix":"10e8331ba1520ebc","entry":"predict_binary","repo":"wiseodd/last_layer_laplace","repo_kind":"official","path":"paper/laplace/util.py","file_url":"https://github.com/wiseodd/last_layer_laplace/blob/HEAD/paper/laplace/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"10e8331ba1520ebc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}