{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deeply-uncertain-comparing-methods-of","title":"Deeply Uncertain: Comparing Methods of Uncertainty Quantification in Deep Learning Algorithms","arxiv_id":"2004.10710","date":"2020-04-22","proceeding":null,"authors":["João Caldeira","Brian Nord"],"abstract":"We present a comparison of methods for uncertainty quantification (UQ) in deep learning algorithms in the context of a simple physical system. Three of the most common uncertainty quantification methods - Bayesian Neural Networks (BNN), Concrete Dropout (CD), and Deep Ensembles (DE) - are compared to the standard analytic error propagation. We discuss this comparison in terms endemic to both machine learning (\"epistemic\" and \"aleatoric\") and the physical sciences (\"statistical\" and \"systematic\"). The comparisons are presented in terms of simulated experimental measurements of a single pendulum - a prototypical physical system for studying measurement and analysis techniques. Our results highlight some pitfalls that may occur when using these UQ methods. For example, when the variation of noise in the training set is small, all methods predicted the same relative uncertainty independently of the inputs. This issue is particularly hard to avoid in BNN. On the other hand, when the test set contains samples far from the training distribution, we found that no methods sufficiently increased the uncertainties associated to their predictions. This problem was particularly clear for CD. In light of these results, we make some recommendations for usage and interpretation of UQ methods.","url_abs":"https://arxiv.org/abs/2004.10710v3","url_pdf":"https://arxiv.org/pdf/2004.10710v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deeply-uncertain-comparing-methods-of","repo_url":"https://github.com/deepskies/DeeplyUncertain-Public","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"uncertainty-quantification","task_name":"Uncertainty Quantification"}],"methods":[{"method_slug":"concrete-dropout","method_name":"Concrete Dropout"},{"method_slug":"deep-ensembles","method_name":"Deep Ensembles"},{"method_slug":"dropout","method_name":"Dropout"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2004.10710","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2004.10710"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deepskies/DeeplyUncertain-Public","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"830613bbdec4cc81","entry":"heteroscedastic_loss","repo":"deepskies/DeeplyUncertain-Public","repo_kind":"official","path":"models/cd.py","file_url":"https://github.com/deepskies/DeeplyUncertain-Public/blob/HEAD/models/cd.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"830613bbdec4cc81"}},{"code_sha256_prefix":"efc992a53400a386","entry":"make_model","repo":"deepskies/DeeplyUncertain-Public","repo_kind":"official","path":"models/cd.py","file_url":"https://github.com/deepskies/DeeplyUncertain-Public/blob/HEAD/models/cd.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"efc992a53400a386"}},{"code_sha256_prefix":"2096dc49ec86fdfc","entry":"mse_loss","repo":"deepskies/DeeplyUncertain-Public","repo_kind":"official","path":"models/cd.py","file_url":"https://github.com/deepskies/DeeplyUncertain-Public/blob/HEAD/models/cd.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2096dc49ec86fdfc"}},{"code_sha256_prefix":"c03137913323b668","entry":"negloglik","repo":"deepskies/DeeplyUncertain-Public","repo_kind":"official","path":"models/mlp_tf.py","file_url":"https://github.com/deepskies/DeeplyUncertain-Public/blob/HEAD/models/mlp_tf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c03137913323b668"}},{"code_sha256_prefix":"c43201ee46e298b3","entry":"pendulum","repo":"deepskies/DeeplyUncertain-Public","repo_kind":"official","path":"data_gen.py","file_url":"https://github.com/deepskies/DeeplyUncertain-Public/blob/HEAD/data_gen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c43201ee46e298b3"}},{"code_sha256_prefix":"5bdad93691882744","entry":"scale","repo":"deepskies/DeeplyUncertain-Public","repo_kind":"official","path":"preprocessing.py","file_url":"https://github.com/deepskies/DeeplyUncertain-Public/blob/HEAD/preprocessing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5bdad93691882744"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}