{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/benchmarking-uncertainty-disentanglement","title":"Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks","arxiv_id":"2402.19460","date":"2024-02-29","proceeding":null,"authors":["Bálint Mucsányi","Michael Kirchhof","Seong Joon Oh"],"abstract":"Uncertainty quantification, once a singular task, has evolved into a spectrum of tasks, including abstained prediction, out-of-distribution detection, and aleatoric uncertainty quantification. The latest goal is disentanglement: the construction of multiple estimators that are each tailored to one and only one source of uncertainty. This paper presents the first benchmark of uncertainty disentanglement. We reimplement and evaluate a comprehensive range of uncertainty estimators, from Bayesian over evidential to deterministic ones, across a diverse range of uncertainty tasks on ImageNet. We find that, despite recent theoretical endeavors, no existing approach provides pairs of disentangled uncertainty estimators in practice. We further find that specialized uncertainty tasks are harder than predictive uncertainty tasks, where we observe saturating performance. Our results provide both practical advice for which uncertainty estimators to use for which specific task, and reveal opportunities for future research toward task-centric and disentangled uncertainties. All our reimplementations and Weights & Biases logs are available at https://github.com/bmucsanyi/untangle.","url_abs":"https://arxiv.org/abs/2402.19460v2","url_pdf":"https://arxiv.org/pdf/2402.19460v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"benchmarking-uncertainty-disentanglement","repo_url":"https://github.com/bmucsanyi/bud","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"benchmarking-uncertainty-disentanglement","repo_url":"https://github.com/bmucsanyi/untangle","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"disentanglement","task_name":"Disentanglement"},{"task_slug":"out-of-distribution-detection","task_name":"Out-of-Distribution Detection"},{"task_slug":"uncertainty-quantification","task_name":"Uncertainty Quantification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.19460","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.19460"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bmucsanyi/bud","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bmucsanyi/untangle","reach":null}],"summary":{"ran":12,"ran_draft_wrong":2,"unverified":5},"by_repo_kind":{"official":{"samples":19,"ran":14,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"39934307038ef5a9","entry":"adaptive_avgmax_pool2d","repo":"bmucsanyi/bud","repo_kind":"official","path":"bud/layers/adaptive_avgmax_pool.py","file_url":"https://github.com/bmucsanyi/bud/blob/HEAD/bud/layers/adaptive_avgmax_pool.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"39934307038ef5a9"}},{"code_sha256_prefix":"fa9e3a5ceeac7703","entry":"adaptive_catavgmax_pool2d","repo":"bmucsanyi/bud","repo_kind":"official","path":"bud/layers/adaptive_avgmax_pool.py","file_url":"https://github.com/bmucsanyi/bud/blob/HEAD/bud/layers/adaptive_avgmax_pool.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fa9e3a5ceeac7703"}},{"code_sha256_prefix":"13295c0de28cc1f1","entry":"adaptive_pool_feat_mult","repo":"bmucsanyi/bud","repo_kind":"official","path":"bud/layers/adaptive_avgmax_pool.py","file_url":"https://github.com/bmucsanyi/bud/blob/HEAD/bud/layers/adaptive_avgmax_pool.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"13295c0de28cc1f1"}},{"code_sha256_prefix":"06cfd20545db1a61","entry":"get_average_metric_values","repo":"bmucsanyi/untangle","repo_kind":"official","path":"validate.py","file_url":"https://github.com/bmucsanyi/untangle/blob/HEAD/validate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"06cfd20545db1a61"}},{"code_sha256_prefix":"816ebe82b6b6275f","entry":"get_condconv_initializer","repo":"bmucsanyi/bud","repo_kind":"official","path":"bud/layers/cond_conv2d.py","file_url":"https://github.com/bmucsanyi/bud/blob/HEAD/bud/layers/cond_conv2d.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"816ebe82b6b6275f"}},{"code_sha256_prefix":"91c27523dde7eb4a","entry":"hard_sigmoid_jit","repo":"bmucsanyi/bud","repo_kind":"official","path":"bud/layers/activations_jit.py","file_url":"https://github.com/bmucsanyi/bud/blob/HEAD/bud/layers/activations_jit.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"91c27523dde7eb4a"}},{"code_sha256_prefix":"801c85b8903bfff5","entry":"mish","repo":"bmucsanyi/bud","repo_kind":"official","path":"bud/layers/activations.py","file_url":"https://github.com/bmucsanyi/bud/blob/HEAD/bud/layers/activations.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"801c85b8903bfff5"}},{"code_sha256_prefix":"fd81feddf7a930d0","entry":"mish_jit","repo":"bmucsanyi/bud","repo_kind":"official","path":"bud/layers/activations_jit.py","file_url":"https://github.com/bmucsanyi/bud/blob/HEAD/bud/layers/activations_jit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fd81feddf7a930d0"}},{"code_sha256_prefix":"d83b6ecf3cdc1d6a","entry":"sigmoid","repo":"bmucsanyi/bud","repo_kind":"official","path":"bud/layers/activations.py","file_url":"https://github.com/bmucsanyi/bud/blob/HEAD/bud/layers/activations.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d83b6ecf3cdc1d6a"}},{"code_sha256_prefix":"c236e1462e2029a7","entry":"swish","repo":"bmucsanyi/bud","repo_kind":"official","path":"bud/layers/activations.py","file_url":"https://github.com/bmucsanyi/bud/blob/HEAD/bud/layers/activations.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c236e1462e2029a7"}},{"code_sha256_prefix":"266ab96375ee74e6","entry":"swish_jit","repo":"bmucsanyi/bud","repo_kind":"official","path":"bud/layers/activations_jit.py","file_url":"https://github.com/bmucsanyi/bud/blob/HEAD/bud/layers/activations_jit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"266ab96375ee74e6"}},{"code_sha256_prefix":"ccf59d4aa2ccacdb","entry":"swish_jit_bwd","repo":"bmucsanyi/bud","repo_kind":"official","path":"bud/layers/activations_me.py","file_url":"https://github.com/bmucsanyi/bud/blob/HEAD/bud/layers/activations_me.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ccf59d4aa2ccacdb"}},{"code_sha256_prefix":"2d53e47b891cd927","entry":"swish_jit_fwd","repo":"bmucsanyi/bud","repo_kind":"official","path":"bud/layers/activations_me.py","file_url":"https://github.com/bmucsanyi/bud/blob/HEAD/bud/layers/activations_me.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2d53e47b891cd927"}},{"code_sha256_prefix":"047f74772af012ee","entry":"swish_me","repo":"bmucsanyi/bud","repo_kind":"official","path":"bud/layers/activations_me.py","file_url":"https://github.com/bmucsanyi/bud/blob/HEAD/bud/layers/activations_me.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"047f74772af012ee"}},{"code_sha256_prefix":"a71eaf07e052ef66","entry":"conv2d_same","repo":"bmucsanyi/bud","repo_kind":"official","path":"bud/layers/conv2d_same.py","file_url":"https://github.com/bmucsanyi/bud/blob/HEAD/bud/layers/conv2d_same.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a71eaf07e052ef66"}},{"code_sha256_prefix":"21e3373d80025972","entry":"create_classifier","repo":"bmucsanyi/bud","repo_kind":"official","path":"bud/layers/classifier.py","file_url":"https://github.com/bmucsanyi/bud/blob/HEAD/bud/layers/classifier.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"21e3373d80025972"}},{"code_sha256_prefix":"475133ca4b96e32e","entry":"create_conv2d_pad","repo":"bmucsanyi/bud","repo_kind":"official","path":"bud/layers/conv2d_same.py","file_url":"https://github.com/bmucsanyi/bud/blob/HEAD/bud/layers/conv2d_same.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"475133ca4b96e32e"}},{"code_sha256_prefix":"62a1f7b2ad0215a1","entry":"rel_logits_1d","repo":"bmucsanyi/bud","repo_kind":"official","path":"bud/layers/bottleneck_attn.py","file_url":"https://github.com/bmucsanyi/bud/blob/HEAD/bud/layers/bottleneck_attn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"62a1f7b2ad0215a1"}},{"code_sha256_prefix":"bf856b7b22366934","entry":"use_fused_attn","repo":"bmucsanyi/bud","repo_kind":"official","path":"bud/layers/config.py","file_url":"https://github.com/bmucsanyi/bud/blob/HEAD/bud/layers/config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bf856b7b22366934"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}