{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/values-a-framework-for-systematic-validation","title":"ValUES: A Framework for Systematic Validation of Uncertainty Estimation in Semantic Segmentation","arxiv_id":"2401.08501","date":"2024-01-16","proceeding":null,"authors":["Kim-Celine Kahl","Carsten T. Lüth","Maximilian Zenk","Klaus Maier-Hein","Paul F. Jaeger"],"abstract":"Uncertainty estimation is an essential and heavily-studied component for the reliable application of semantic segmentation methods. While various studies exist claiming methodological advances on the one hand, and successful application on the other hand, the field is currently hampered by a gap between theory and practice leaving fundamental questions unanswered: Can data-related and model-related uncertainty really be separated in practice? Which components of an uncertainty method are essential for real-world performance? Which uncertainty method works well for which application? In this work, we link this research gap to a lack of systematic and comprehensive evaluation of uncertainty methods. Specifically, we identify three key pitfalls in current literature and present an evaluation framework that bridges the research gap by providing 1) a controlled environment for studying data ambiguities as well as distribution shifts, 2) systematic ablations of relevant method components, and 3) test-beds for the five predominant uncertainty applications: OoD-detection, active learning, failure detection, calibration, and ambiguity modeling. Empirical results on simulated as well as real-world data demonstrate how the proposed framework is able to answer the predominant questions in the field revealing for instance that 1) separation of uncertainty types works on simulated data but does not necessarily translate to real-world data, 2) aggregation of scores is a crucial but currently neglected component of uncertainty methods, 3) While ensembles are performing most robustly across the different downstream tasks and settings, test-time augmentation often constitutes a light-weight alternative. Code is at: https://github.com/IML-DKFZ/values","url_abs":"https://arxiv.org/abs/2401.08501v2","url_pdf":"https://arxiv.org/pdf/2401.08501v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"values-a-framework-for-systematic-validation","repo_url":"https://github.com/iml-dkfz/values","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"active-learning","task_name":"Active Learning"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.08501","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.08501"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/IML-DKFZ/values","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/iml-dkfz/values","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"ran":6,"unverified":3},"by_repo_kind":{"official":{"samples":10,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fac5364e2f53c6db","entry":"conv3x3","repo":"IML-DKFZ/values","repo_kind":"official","path":"uncertainty_modeling/models/hrnet_module.py","file_url":"https://github.com/IML-DKFZ/values/blob/HEAD/uncertainty_modeling/models/hrnet_module.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fac5364e2f53c6db"}},{"code_sha256_prefix":"e5c66f25e43363f2","entry":"get_train_data_samples","repo":"IML-DKFZ/values","repo_kind":"official","path":"uncertainty_modeling/lidc_idri_datamodule_3D.py","file_url":"https://github.com/IML-DKFZ/values/blob/HEAD/uncertainty_modeling/lidc_idri_datamodule_3D.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e5c66f25e43363f2"}},{"code_sha256_prefix":"efa971e5017cd37b","entry":"get_train_data_samples","repo":"IML-DKFZ/values","repo_kind":"official","path":"uncertainty_modeling/toy_datamodule_3D.py","file_url":"https://github.com/IML-DKFZ/values/blob/HEAD/uncertainty_modeling/toy_datamodule_3D.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"efa971e5017cd37b"}},{"code_sha256_prefix":"b10eee998c14b87a","entry":"get_val_test_data_samples","repo":"IML-DKFZ/values","repo_kind":"official","path":"uncertainty_modeling/lidc_idri_datamodule_3D.py","file_url":"https://github.com/IML-DKFZ/values/blob/HEAD/uncertainty_modeling/lidc_idri_datamodule_3D.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b10eee998c14b87a"}},{"code_sha256_prefix":"ce47586ffdff2741","entry":"get_val_test_data_samples","repo":"IML-DKFZ/values","repo_kind":"official","path":"uncertainty_modeling/toy_datamodule_3D.py","file_url":"https://github.com/IML-DKFZ/values/blob/HEAD/uncertainty_modeling/toy_datamodule_3D.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ce47586ffdff2741"}},{"code_sha256_prefix":"5775655dbdd56e51","entry":"soft_dice","repo":"IML-DKFZ/values","repo_kind":"official","path":"uncertainty_modeling/loss_modules.py","file_url":"https://github.com/IML-DKFZ/values/blob/HEAD/uncertainty_modeling/loss_modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5775655dbdd56e51"}},{"code_sha256_prefix":"03cb8b894670bbd8","entry":"subfiles","repo":"IML-DKFZ/values","repo_kind":"official","path":"uncertainty_modeling/lidc_idri_datamodule_3D.py","file_url":"https://github.com/IML-DKFZ/values/blob/HEAD/uncertainty_modeling/lidc_idri_datamodule_3D.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"03cb8b894670bbd8"}},{"code_sha256_prefix":"42ad3b72fe54e6f3","entry":"SsnUNet3D","repo":"iml-dkfz/values","repo_kind":"official","path":"uncertainty_modeling/models/ssn_unet3D_module.py","file_url":"https://github.com/iml-dkfz/values/blob/HEAD/uncertainty_modeling/models/ssn_unet3D_module.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42ad3b72fe54e6f3"}},{"code_sha256_prefix":"48c5ca3e2ad0db57","entry":"UNet3D","repo":"iml-dkfz/values","repo_kind":"official","path":"uncertainty_modeling/models/ssn_unet3D_module.py","file_url":"https://github.com/iml-dkfz/values/blob/HEAD/uncertainty_modeling/models/ssn_unet3D_module.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"48c5ca3e2ad0db57"}},{"code_sha256_prefix":"f47754874c0be188","entry":"get_seg_model","repo":"IML-DKFZ/values","repo_kind":"official","path":"uncertainty_modeling/models/hrnet_module.py","file_url":"https://github.com/IML-DKFZ/values/blob/HEAD/uncertainty_modeling/models/hrnet_module.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f47754874c0be188"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}