{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2601-17103","title":"Performance uncertainty in medical image analysis: a large-scale investigation of confidence intervals","arxiv_id":"2601.17103","date":"2026-01-23","proceeding":null,"authors":["Pascaline André","Charles Heitz","Evangelia Christodoulou","Annika Reinke","Carole H. Sudre","Michela Antonelli","Patrick Godau","M. Jorge Cardoso","Antoine Gilson","Sophie Tezenas du Montcel","Gaël Varoquaux","Lena Maier-Hein","Olivier Colliot"],"abstract":"Performance uncertainty quantification is essential for reliable validation and eventual clinical translation of medical imaging artificial intelligence (AI). Confidence intervals (CIs) play a central role in this process by indicating how precise a reported performance estimate is. Yet, due to the limited amount of work examining CI behavior in medical imaging, the community remains largely unaware of how many diverse CI methods exist and how they behave in specific settings. The purpose of this study is to close this gap. To this end, we conducted a large-scale empirical analysis across a total of 24 segmentation and classification tasks, using 19 trained models per task group, a broad spectrum of commonly used performance metrics, multiple aggregation strategies, and several widely adopted CI methods. Reliability (coverage) and precision (width) of each CI method were estimated across all settings to characterize their dependence on study characteristics. Our analysis revealed five principal findings: 1) the sample size required for reliable CIs varies from a few dozens to several thousands of cases depending on study parameters; 2) CI behavior is strongly affected by the choice of performance metric; 3) aggregation strategy substantially influences the reliability of CIs, e.g. they require more observations for macro than for micro; 4) the machine learning problem (segmentation versus classification) modulates these effects; 5) different CI methods are not equally reliable and precise depending on the use case. Finally, we derived practical implications of this study in the form of a decision tree which shall prove useful to the community. This paves the way for future consensus guidelines on reporting performance uncertainty.","url_abs":"https://arxiv.org/abs/2601.17103","url_pdf":"https://arxiv.org/pdf/2601.17103","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2601.17103"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/aramis-lab/CIs_Medical_Imaging","reach":null},{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/aramis-lab/CIs_Medical_Imaging_Plots","reach":{"status":"ok"}}],"summary":{"unverified":24},"by_repo_kind":{"found_in_text":{"samples":24,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":24,"samples":[{"code_sha256_prefix":"14dfb3060a0bec47","entry":"accuracy","repo":"aramis-lab/CIs_Medical_Imaging","repo_kind":"found_in_text","path":"src/intervals_and_metrics/pixel_wise_metrics.py","file_url":"https://github.com/aramis-lab/CIs_Medical_Imaging/blob/HEAD/src/intervals_and_metrics/pixel_wise_metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"14dfb3060a0bec47"}},{"code_sha256_prefix":"0385669db116ea5d","entry":"compute_CIs_segmentation","repo":"aramis-lab/CIs_Medical_Imaging","repo_kind":"found_in_text","path":"src/intervals_and_metrics/intervals_segm.py","file_url":"https://github.com/aramis-lab/CIs_Medical_Imaging/blob/HEAD/src/intervals_and_metrics/intervals_segm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0385669db116ea5d"}},{"code_sha256_prefix":"0d7de4f11447d615","entry":"compute_scaled_differences","repo":"aramis-lab/CIs_Medical_Imaging","repo_kind":"found_in_text","path":"src/kernels/kernels_preprocessing_utils.py","file_url":"https://github.com/aramis-lab/CIs_Medical_Imaging/blob/HEAD/src/kernels/kernels_preprocessing_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0d7de4f11447d615"}},{"code_sha256_prefix":"e0036700747949bc","entry":"cosine_kernel","repo":"aramis-lab/CIs_Medical_Imaging","repo_kind":"found_in_text","path":"src/kernels/cosine.py","file_url":"https://github.com/aramis-lab/CIs_Medical_Imaging/blob/HEAD/src/kernels/cosine.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e0036700747949bc"}},{"code_sha256_prefix":"3d63aa0afddb2983","entry":"epanechnikov_kernel","repo":"aramis-lab/CIs_Medical_Imaging","repo_kind":"found_in_text","path":"src/kernels/epanechnikov.py","file_url":"https://github.com/aramis-lab/CIs_Medical_Imaging/blob/HEAD/src/kernels/epanechnikov.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3d63aa0afddb2983"}},{"code_sha256_prefix":"fac0fddf035369bb","entry":"exponential_kernel","repo":"aramis-lab/CIs_Medical_Imaging","repo_kind":"found_in_text","path":"src/kernels/exponential.py","file_url":"https://github.com/aramis-lab/CIs_Medical_Imaging/blob/HEAD/src/kernels/exponential.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fac0fddf035369bb"}},{"code_sha256_prefix":"73016539ac8a3be7","entry":"gaussian_kernel","repo":"aramis-lab/CIs_Medical_Imaging","repo_kind":"found_in_text","path":"src/kernels/gaussian.py","file_url":"https://github.com/aramis-lab/CIs_Medical_Imaging/blob/HEAD/src/kernels/gaussian.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"73016539ac8a3be7"}},{"code_sha256_prefix":"1218603dbe830903","entry":"get_authorized_methods_classification","repo":"aramis-lab/CIs_Medical_Imaging","repo_kind":"found_in_text","path":"src/intervals_and_metrics/authorized_methods.py","file_url":"https://github.com/aramis-lab/CIs_Medical_Imaging/blob/HEAD/src/intervals_and_metrics/authorized_methods.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1218603dbe830903"}},{"code_sha256_prefix":"268cc73fcca7375e","entry":"get_authorized_methods_segmentation","repo":"aramis-lab/CIs_Medical_Imaging","repo_kind":"found_in_text","path":"src/intervals_and_metrics/authorized_methods.py","file_url":"https://github.com/aramis-lab/CIs_Medical_Imaging/blob/HEAD/src/intervals_and_metrics/authorized_methods.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"268cc73fcca7375e"}},{"code_sha256_prefix":"1af7683d986baffa","entry":"get_bounds","repo":"aramis-lab/CIs_Medical_Imaging","repo_kind":"found_in_text","path":"src/intervals_and_metrics/metrics_bounds_and_continuity.py","file_url":"https://github.com/aramis-lab/CIs_Medical_Imaging/blob/HEAD/src/intervals_and_metrics/metrics_bounds_and_continuity.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1af7683d986baffa"}},{"code_sha256_prefix":"e8c5237bf095eb5a","entry":"is_continuous","repo":"aramis-lab/CIs_Medical_Imaging","repo_kind":"found_in_text","path":"src/intervals_and_metrics/metrics_bounds_and_continuity.py","file_url":"https://github.com/aramis-lab/CIs_Medical_Imaging/blob/HEAD/src/intervals_and_metrics/metrics_bounds_and_continuity.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e8c5237bf095eb5a"}},{"code_sha256_prefix":"5775b8968033563c","entry":"label_binarize_vectorized","repo":"aramis-lab/CIs_Medical_Imaging","repo_kind":"found_in_text","path":"src/intervals_and_metrics/pixel_wise_metrics.py","file_url":"https://github.com/aramis-lab/CIs_Medical_Imaging/blob/HEAD/src/intervals_and_metrics/pixel_wise_metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5775b8968033563c"}},{"code_sha256_prefix":"010f208d071aca12","entry":"load_config","repo":"aramis-lab/CIs_Medical_Imaging","repo_kind":"found_in_text","path":"src/run_single.py","file_url":"https://github.com/aramis-lab/CIs_Medical_Imaging/blob/HEAD/src/run_single.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"010f208d071aca12"}},{"code_sha256_prefix":"44e27bcadb51c86d","entry":"param_t_interval","repo":"aramis-lab/CIs_Medical_Imaging","repo_kind":"found_in_text","path":"src/intervals_and_metrics/intervals_segm.py","file_url":"https://github.com/aramis-lab/CIs_Medical_Imaging/blob/HEAD/src/intervals_and_metrics/intervals_segm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"44e27bcadb51c86d"}},{"code_sha256_prefix":"982066b798d08f0f","entry":"param_z_interval","repo":"aramis-lab/CIs_Medical_Imaging","repo_kind":"found_in_text","path":"src/intervals_and_metrics/intervals_segm.py","file_url":"https://github.com/aramis-lab/CIs_Medical_Imaging/blob/HEAD/src/intervals_and_metrics/intervals_segm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"982066b798d08f0f"}},{"code_sha256_prefix":"86a325366306dbe2","entry":"preprocess_kernel_inputs","repo":"aramis-lab/CIs_Medical_Imaging","repo_kind":"found_in_text","path":"src/kernels/kernels_preprocessing_utils.py","file_url":"https://github.com/aramis-lab/CIs_Medical_Imaging/blob/HEAD/src/kernels/kernels_preprocessing_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"86a325366306dbe2"}},{"code_sha256_prefix":"bc80eca4f17af58a","entry":"sample_cosine_multivariate","repo":"aramis-lab/CIs_Medical_Imaging","repo_kind":"found_in_text","path":"src/kernels/cosine.py","file_url":"https://github.com/aramis-lab/CIs_Medical_Imaging/blob/HEAD/src/kernels/cosine.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bc80eca4f17af58a"}},{"code_sha256_prefix":"1898612c4f157665","entry":"sample_epanechnikov_multivariate","repo":"aramis-lab/CIs_Medical_Imaging","repo_kind":"found_in_text","path":"src/kernels/epanechnikov.py","file_url":"https://github.com/aramis-lab/CIs_Medical_Imaging/blob/HEAD/src/kernels/epanechnikov.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1898612c4f157665"}},{"code_sha256_prefix":"e92a5b1eec342004","entry":"sample_exponential_multivariate","repo":"aramis-lab/CIs_Medical_Imaging","repo_kind":"found_in_text","path":"src/kernels/exponential.py","file_url":"https://github.com/aramis-lab/CIs_Medical_Imaging/blob/HEAD/src/kernels/exponential.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e92a5b1eec342004"}},{"code_sha256_prefix":"2d764936c34ea357","entry":"sample_gaussian_multivariate","repo":"aramis-lab/CIs_Medical_Imaging","repo_kind":"found_in_text","path":"src/kernels/gaussian.py","file_url":"https://github.com/aramis-lab/CIs_Medical_Imaging/blob/HEAD/src/kernels/gaussian.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2d764936c34ea357"}},{"code_sha256_prefix":"b46968b82a0b751c","entry":"sample_weighted_kde","repo":"aramis-lab/CIs_Medical_Imaging","repo_kind":"found_in_text","path":"src/kde/weighted_adaptive_kde.py","file_url":"https://github.com/aramis-lab/CIs_Medical_Imaging/blob/HEAD/src/kde/weighted_adaptive_kde.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b46968b82a0b751c"}},{"code_sha256_prefix":"c96f52933b13bbe9","entry":"sample_weighted_kde_multivariate","repo":"aramis-lab/CIs_Medical_Imaging","repo_kind":"found_in_text","path":"src/kde/weighted_adaptive_kde.py","file_url":"https://github.com/aramis-lab/CIs_Medical_Imaging/blob/HEAD/src/kde/weighted_adaptive_kde.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c96f52933b13bbe9"}},{"code_sha256_prefix":"8cc426ecde08301f","entry":"softmax","repo":"aramis-lab/CIs_Medical_Imaging","repo_kind":"found_in_text","path":"src/intervals_and_metrics/pixel_wise_metrics.py","file_url":"https://github.com/aramis-lab/CIs_Medical_Imaging/blob/HEAD/src/intervals_and_metrics/pixel_wise_metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8cc426ecde08301f"}},{"code_sha256_prefix":"7a7bef604e497c5c","entry":"weighted_kde","repo":"aramis-lab/CIs_Medical_Imaging","repo_kind":"found_in_text","path":"src/kde/weighted_adaptive_kde.py","file_url":"https://github.com/aramis-lab/CIs_Medical_Imaging/blob/HEAD/src/kde/weighted_adaptive_kde.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7a7bef604e497c5c"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}