{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/uncertainty-toolbox-an-open-source-library","title":"Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification","arxiv_id":"2109.10254","date":"2021-09-21","proceeding":null,"authors":["Youngseog Chung","Ian Char","Han Guo","Jeff Schneider","Willie Neiswanger"],"abstract":"With increasing deployment of machine learning systems in various real-world tasks, there is a greater need for accurate quantification of predictive uncertainty. While the common goal in uncertainty quantification (UQ) in machine learning is to approximate the true distribution of the target data, many works in UQ tend to be disjoint in the evaluation metrics utilized, and disparate implementations for each metric lead to numerical results that are not directly comparable across different works. To address this, we introduce Uncertainty Toolbox, an open-source python library that helps to assess, visualize, and improve UQ. Uncertainty Toolbox additionally provides pedagogical resources, such as a glossary of key terms and an organized collection of key paper references. We hope that this toolbox is useful for accelerating and uniting research efforts in uncertainty in machine learning.","url_abs":"https://arxiv.org/abs/2109.10254v1","url_pdf":"https://arxiv.org/pdf/2109.10254v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"uncertainty-toolbox-an-open-source-library","repo_url":"https://github.com/uncertainty-toolbox/uncertainty-toolbox","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"uncertainty-toolbox-an-open-source-library","repo_url":"https://github.com/DILiS-lab/drivers-of-predictive-aleatoric-uncertainty","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"CC-BY-SA-4.0"}}],"tasks":[{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"},{"task_slug":"uncertainty-quantification","task_name":"Uncertainty Quantification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2109.10254","atlas_url":"https://app.syntology.ai/?focus=2109.10254","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2109.10254"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/DILiS-lab/drivers-of-predictive-aleatoric-uncertainty","reach":{"status":"ok","spdx":"CC-BY-SA-4.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/uncertainty-toolbox/uncertainty-toolbox","reach":null}],"summary":{"ran_fixture":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fc0d6fe304a9300e","entry":"plot_intervals","repo":"uncertainty-toolbox/uncertainty-toolbox","repo_kind":"official","path":"uncertainty_toolbox/viz.py","file_url":"https://github.com/uncertainty-toolbox/uncertainty-toolbox/blob/HEAD/uncertainty_toolbox/viz.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fc0d6fe304a9300e"}},{"code_sha256_prefix":"c91b9d1ff6fd6336","entry":"plot_intervals_ordered","repo":"uncertainty-toolbox/uncertainty-toolbox","repo_kind":"official","path":"uncertainty_toolbox/viz.py","file_url":"https://github.com/uncertainty-toolbox/uncertainty-toolbox/blob/HEAD/uncertainty_toolbox/viz.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c91b9d1ff6fd6336"}},{"code_sha256_prefix":"cb5c16f07da4476c","entry":"plot_xy","repo":"uncertainty-toolbox/uncertainty-toolbox","repo_kind":"official","path":"uncertainty_toolbox/viz.py","file_url":"https://github.com/uncertainty-toolbox/uncertainty-toolbox/blob/HEAD/uncertainty_toolbox/viz.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cb5c16f07da4476c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}