{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/conformal-performance-range-prediction-for","title":"Conformal Performance Range Prediction for Segmentation Output Quality Control","arxiv_id":"2407.13307","date":"2024-07-18","proceeding":null,"authors":["Anna M. Wundram","Paul Fischer","Michael Muehlebach","Lisa M. Koch","Christian F. Baumgartner"],"abstract":"Recent works have introduced methods to estimate segmentation performance without ground truth, relying solely on neural network softmax outputs. These techniques hold potential for intuitive output quality control. However, such performance estimates rely on calibrated softmax outputs, which is often not the case in modern neural networks. Moreover, the estimates do not take into account inherent uncertainty in segmentation tasks. These limitations may render precise performance predictions unattainable, restricting the practical applicability of performance estimation methods. To address these challenges, we develop a novel approach for predicting performance ranges with statistical guarantees of containing the ground truth with a user specified probability. Our method leverages sampling-based segmentation uncertainty estimation to derive heuristic performance ranges, and applies split conformal prediction to transform these estimates into rigorous prediction ranges that meet the desired guarantees. We demonstrate our approach on the FIVES retinal vessel segmentation dataset and compare five commonly used sampling-based uncertainty estimation techniques. Our results show that it is possible to achieve the desired coverage with small prediction ranges, highlighting the potential of performance range prediction as a valuable tool for output quality control.","url_abs":"https://arxiv.org/abs/2407.13307v2","url_pdf":"https://arxiv.org/pdf/2407.13307v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"conformal-performance-range-prediction-for","repo_url":"https://github.com/annawundram/PerformanceRangePrediction","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"conformal-prediction","task_name":"Conformal Prediction"},{"task_slug":"prediction","task_name":"Prediction"},{"task_slug":"retinal-vessel-segmentation","task_name":"Retinal Vessel Segmentation"},{"task_slug":"segmentation","task_name":"Segmentation"}],"methods":[{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.13307","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.13307"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/annawundram/PerformanceRangePrediction","reach":null}],"summary":{"ran_violates":2,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"951c1e7e4145f7d5","entry":"dice_coefficient","repo":"annawundram/PerformanceRangePrediction","repo_kind":"official","path":"PerformancePrediction.py","file_url":"https://github.com/annawundram/PerformanceRangePrediction/blob/HEAD/PerformancePrediction.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"951c1e7e4145f7d5"}},{"code_sha256_prefix":"6ad8a6d1828b706b","entry":"predict_dice","repo":"annawundram/PerformanceRangePrediction","repo_kind":"official","path":"PerformancePrediction.py","file_url":"https://github.com/annawundram/PerformanceRangePrediction/blob/HEAD/PerformancePrediction.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6ad8a6d1828b706b"}},{"code_sha256_prefix":"c471b12d586d0dc4","entry":"get_samples","repo":"annawundram/PerformanceRangePrediction","repo_kind":"official","path":"PerformancePrediction.py","file_url":"https://github.com/annawundram/PerformanceRangePrediction/blob/HEAD/PerformancePrediction.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c471b12d586d0dc4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}