{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/uncertainty-baselines-benchmarks-for","title":"Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning","arxiv_id":"2106.04015","date":"2021-06-07","proceeding":null,"authors":["Zachary Nado","Neil Band","Mark Collier","Josip Djolonga","Michael W. Dusenberry","Sebastian Farquhar","Qixuan Feng","Angelos Filos","Marton Havasi","Rodolphe Jenatton","Ghassen Jerfel","Jeremiah Liu","Zelda Mariet","Jeremy Nixon","Shreyas Padhy","Jie Ren","Tim G. J. Rudner","Faris Sbahi","Yeming Wen","Florian Wenzel","Kevin Murphy","D. Sculley","Balaji Lakshminarayanan","Jasper Snoek","Yarin Gal","Dustin Tran"],"abstract":"High-quality estimates of uncertainty and robustness are crucial for numerous real-world applications, especially for deep learning which underlies many deployed ML systems. The ability to compare techniques for improving these estimates is therefore very important for research and practice alike. Yet, competitive comparisons of methods are often lacking due to a range of reasons, including: compute availability for extensive tuning, incorporation of sufficiently many baselines, and concrete documentation for reproducibility. In this paper we introduce Uncertainty Baselines: high-quality implementations of standard and state-of-the-art deep learning methods on a variety of tasks. As of this writing, the collection spans 19 methods across 9 tasks, each with at least 5 metrics. Each baseline is a self-contained experiment pipeline with easily reusable and extendable components. Our goal is to provide immediate starting points for experimentation with new methods or applications. Additionally we provide model checkpoints, experiment outputs as Python notebooks, and leaderboards for comparing results. Code available at https://github.com/google/uncertainty-baselines.","url_abs":"https://arxiv.org/abs/2106.04015v3","url_pdf":"https://arxiv.org/pdf/2106.04015v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"uncertainty-baselines-benchmarks-for","repo_url":"https://github.com/google/uncertainty-baselines","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"uncertainty-baselines-benchmarks-for","repo_url":"https://github.com/angie-lab-jhu/density_softmax","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"uncertainty-baselines-benchmarks-for","repo_url":"https://github.com/thlarsen13/thlarsen_uncertainty_models","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"deep-learning","task_name":"Deep Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.04015","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.04015"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google/uncertainty-baselines","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/angie-lab-jhu/density_softmax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thlarsen13/thlarsen_uncertainty_models","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2,"ran_honours":1,"unverified":8},"by_repo_kind":{"listed":{"samples":11,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"295ee5b09ae2c97c","entry":"create_config","repo":"thlarsen13/thlarsen_uncertainty_models","repo_kind":"listed","path":"uncertainty_baselines/models/bert.py","file_url":"https://github.com/thlarsen13/thlarsen_uncertainty_models/blob/HEAD/uncertainty_baselines/models/bert.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"295ee5b09ae2c97c"}},{"code_sha256_prefix":"40ffce7d1720bb3e","entry":"create_feature_and_label","repo":"thlarsen13/thlarsen_uncertainty_models","repo_kind":"listed","path":"uncertainty_baselines/models/bert.py","file_url":"https://github.com/thlarsen13/thlarsen_uncertainty_models/blob/HEAD/uncertainty_baselines/models/bert.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"40ffce7d1720bb3e"}},{"code_sha256_prefix":"9e354d875b0e94e6","entry":"efficientnet_params","repo":"thlarsen13/thlarsen_uncertainty_models","repo_kind":"listed","path":"uncertainty_baselines/models/efficientnet.py","file_url":"https://github.com/thlarsen13/thlarsen_uncertainty_models/blob/HEAD/uncertainty_baselines/models/efficientnet.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9e354d875b0e94e6"}},{"code_sha256_prefix":"eff25194ff1a4c16","entry":"conv_kernel_initializer","repo":"thlarsen13/thlarsen_uncertainty_models","repo_kind":"listed","path":"uncertainty_baselines/models/efficientnet_utils.py","file_url":"https://github.com/thlarsen13/thlarsen_uncertainty_models/blob/HEAD/uncertainty_baselines/models/efficientnet_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eff25194ff1a4c16"}},{"code_sha256_prefix":"48ffc519251345a8","entry":"drop_connect","repo":"thlarsen13/thlarsen_uncertainty_models","repo_kind":"listed","path":"uncertainty_baselines/models/efficientnet_utils.py","file_url":"https://github.com/thlarsen13/thlarsen_uncertainty_models/blob/HEAD/uncertainty_baselines/models/efficientnet_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"48ffc519251345a8"}},{"code_sha256_prefix":"bc1878c9e235eca0","entry":"efficientnet_params","repo":"thlarsen13/thlarsen_uncertainty_models","repo_kind":"listed","path":"uncertainty_baselines/models/efficientnet_utils.py","file_url":"https://github.com/thlarsen13/thlarsen_uncertainty_models/blob/HEAD/uncertainty_baselines/models/efficientnet_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bc1878c9e235eca0"}},{"code_sha256_prefix":"4979d2806a89ebe3","entry":"genomics_cnn","repo":"thlarsen13/thlarsen_uncertainty_models","repo_kind":"listed","path":"uncertainty_baselines/models/genomics_cnn.py","file_url":"https://github.com/thlarsen13/thlarsen_uncertainty_models/blob/HEAD/uncertainty_baselines/models/genomics_cnn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4979d2806a89ebe3"}},{"code_sha256_prefix":"7b2943bee6656eca","entry":"get_adjacency_matrix","repo":"thlarsen13/thlarsen_uncertainty_models","repo_kind":"listed","path":"uncertainty_baselines/models/mpnn.py","file_url":"https://github.com/thlarsen13/thlarsen_uncertainty_models/blob/HEAD/uncertainty_baselines/models/mpnn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7b2943bee6656eca"}},{"code_sha256_prefix":"38f23256728e342b","entry":"movielens","repo":"thlarsen13/thlarsen_uncertainty_models","repo_kind":"listed","path":"uncertainty_baselines/models/movielens.py","file_url":"https://github.com/thlarsen13/thlarsen_uncertainty_models/blob/HEAD/uncertainty_baselines/models/movielens.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"38f23256728e342b"}},{"code_sha256_prefix":"ec0ebc77bce35a4a","entry":"round_filters","repo":"thlarsen13/thlarsen_uncertainty_models","repo_kind":"listed","path":"uncertainty_baselines/models/efficientnet.py","file_url":"https://github.com/thlarsen13/thlarsen_uncertainty_models/blob/HEAD/uncertainty_baselines/models/efficientnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ec0ebc77bce35a4a"}},{"code_sha256_prefix":"e316074dac020280","entry":"round_repeats","repo":"thlarsen13/thlarsen_uncertainty_models","repo_kind":"listed","path":"uncertainty_baselines/models/efficientnet.py","file_url":"https://github.com/thlarsen13/thlarsen_uncertainty_models/blob/HEAD/uncertainty_baselines/models/efficientnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e316074dac020280"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}