{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/conformal-prediction-for-class-wise-coverage","title":"Conformal Prediction for Class-wise Coverage via Augmented Label Rank Calibration","arxiv_id":"2406.06818","date":"2024-06-10","proceeding":null,"authors":["Yuanjie Shi","Subhankar Ghosh","Taha Belkhouja","Janardhan Rao Doppa","Yan Yan"],"abstract":"Conformal prediction (CP) is an emerging uncertainty quantification framework that allows us to construct a prediction set to cover the true label with a pre-specified marginal or conditional probability. Although the valid coverage guarantee has been extensively studied for classification problems, CP often produces large prediction sets which may not be practically useful. This issue is exacerbated for the setting of class-conditional coverage on imbalanced classification tasks with many and/or imbalanced classes. This paper proposes the Rank Calibrated Class-conditional CP (RC3P) algorithm to reduce the prediction set sizes to achieve class-conditional coverage, where the valid coverage holds for each class. In contrast to the standard class-conditional CP (CCP) method that uniformly thresholds the class-wise conformity score for each class, the augmented label rank calibration step allows RC3P to selectively iterate this class-wise thresholding subroutine only for a subset of classes whose class-wise top-k error is small. We prove that agnostic to the classifier and data distribution, RC3P achieves class-wise coverage. We also show that RC3P reduces the size of prediction sets compared to the CCP method. Comprehensive experiments on multiple real-world datasets demonstrate that RC3P achieves class-wise coverage and 26.25% reduction in prediction set sizes on average.","url_abs":"https://arxiv.org/abs/2406.06818v5","url_pdf":"https://arxiv.org/pdf/2406.06818v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"conformal-prediction-for-class-wise-coverage","repo_url":"https://github.com/yuanjiesh/rc3p","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"conformal-prediction","task_name":"Conformal Prediction"},{"task_slug":"prediction","task_name":"Prediction"},{"task_slug":"uncertainty-quantification","task_name":"Uncertainty Quantification"},{"task_slug":"imbalanced-classification","task_name":"imbalanced classification"},{"task_slug":null,"task_name":"valid"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.06818","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.06818"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yuanjiesh/rc3p","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":10,"ran_draft_wrong":1,"unverified":13},"by_repo_kind":{"official":{"samples":24,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a13fb468b655e0b0","entry":"Extract","repo":"yuanjiesh/rc3p","repo_kind":"official","path":"ICP/utils_others.py","file_url":"https://github.com/yuanjiesh/rc3p/blob/HEAD/ICP/utils_others.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a13fb468b655e0b0"}},{"code_sha256_prefix":"bf0624cf01d64c5b","entry":"average_results_across_seeds","repo":"yuanjiesh/rc3p","repo_kind":"official","path":"ICP/main_aps.py","file_url":"https://github.com/yuanjiesh/rc3p/blob/HEAD/ICP/main_aps.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bf0624cf01d64c5b"}},{"code_sha256_prefix":"8de2541ba434c627","entry":"embed_all_classes","repo":"yuanjiesh/rc3p","repo_kind":"official","path":"ICP/clustering_utils.py","file_url":"https://github.com/yuanjiesh/rc3p/blob/HEAD/ICP/clustering_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8de2541ba434c627"}},{"code_sha256_prefix":"4dcf06ba28983266","entry":"focal_loss","repo":"yuanjiesh/rc3p","repo_kind":"official","path":"train/losses.py","file_url":"https://github.com/yuanjiesh/rc3p/blob/HEAD/train/losses.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4dcf06ba28983266"}},{"code_sha256_prefix":"ec757d7558e850db","entry":"generate_synthetic_clustered_data","repo":"yuanjiesh/rc3p","repo_kind":"official","path":"ICP/clustering_utils.py","file_url":"https://github.com/yuanjiesh/rc3p/blob/HEAD/ICP/clustering_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec757d7558e850db"}},{"code_sha256_prefix":"684de5d3a8583399","entry":"get_quantile_threshold","repo":"yuanjiesh/rc3p","repo_kind":"official","path":"ICP/conformal_utils2.py","file_url":"https://github.com/yuanjiesh/rc3p/blob/HEAD/ICP/conformal_utils2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"684de5d3a8583399"}},{"code_sha256_prefix":"652a22184abebb5b","entry":"initialize_dict","repo":"yuanjiesh/rc3p","repo_kind":"official","path":"ICP/main_aps.py","file_url":"https://github.com/yuanjiesh/rc3p/blob/HEAD/ICP/main_aps.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"652a22184abebb5b"}},{"code_sha256_prefix":"0d90c4b9c588c0bd","entry":"initialize_metrics_dict","repo":"yuanjiesh/rc3p","repo_kind":"official","path":"ICP/main_aps.py","file_url":"https://github.com/yuanjiesh/rc3p/blob/HEAD/ICP/main_aps.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d90c4b9c588c0bd"}},{"code_sha256_prefix":"0d4621927b1a4c0f","entry":"quantile_embedding","repo":"yuanjiesh/rc3p","repo_kind":"official","path":"ICP/clustering_utils.py","file_url":"https://github.com/yuanjiesh/rc3p/blob/HEAD/ICP/clustering_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d4621927b1a4c0f"}},{"code_sha256_prefix":"d3fbca0bd0a89dc5","entry":"random_split","repo":"yuanjiesh/rc3p","repo_kind":"official","path":"ICP/conformal_utils2.py","file_url":"https://github.com/yuanjiesh/rc3p/blob/HEAD/ICP/conformal_utils2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d3fbca0bd0a89dc5"}},{"code_sha256_prefix":"3293a3745a74653c","entry":"split_X_and_y_Orin","repo":"yuanjiesh/rc3p","repo_kind":"official","path":"ICP/conformal_utils2.py","file_url":"https://github.com/yuanjiesh/rc3p/blob/HEAD/ICP/conformal_utils2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3293a3745a74653c"}},{"code_sha256_prefix":"df0fd09f60a64497","entry":"FormData","repo":"yuanjiesh/rc3p","repo_kind":"official","path":"ICP/utils_others.py","file_url":"https://github.com/yuanjiesh/rc3p/blob/HEAD/ICP/utils_others.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"df0fd09f60a64497"}},{"code_sha256_prefix":"165c9590c92e284d","entry":"accuracy_matrix","repo":"yuanjiesh/rc3p","repo_kind":"official","path":"ICP/generate_score.py","file_url":"https://github.com/yuanjiesh/rc3p/blob/HEAD/ICP/generate_score.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"165c9590c92e284d"}},{"code_sha256_prefix":"5e94402af51c3024","entry":"calc_top_k_accuracy_per_class","repo":"yuanjiesh/rc3p","repo_kind":"official","path":"ICP/generate_score.py","file_url":"https://github.com/yuanjiesh/rc3p/blob/HEAD/ICP/generate_score.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e94402af51c3024"}},{"code_sha256_prefix":"d20cc174be4ee128","entry":"get_logits_targets","repo":"yuanjiesh/rc3p","repo_kind":"official","path":"ICP/utils_others.py","file_url":"https://github.com/yuanjiesh/rc3p/blob/HEAD/ICP/utils_others.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d20cc174be4ee128"}},{"code_sha256_prefix":"e156cd71262c686a","entry":"get_softmax_and_ranks","repo":"yuanjiesh/rc3p","repo_kind":"official","path":"ICP/generate_score.py","file_url":"https://github.com/yuanjiesh/rc3p/blob/HEAD/ICP/generate_score.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e156cd71262c686a"}},{"code_sha256_prefix":"e6aa9494b00ba140","entry":"load_Emnist","repo":"yuanjiesh/rc3p","repo_kind":"official","path":"dataset/EMNIST.py","file_url":"https://github.com/yuanjiesh/rc3p/blob/HEAD/dataset/EMNIST.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e6aa9494b00ba140"}},{"code_sha256_prefix":"f2b5240bcfd4cf3a","entry":"resnet18","repo":"yuanjiesh/rc3p","repo_kind":"official","path":"models/resnet_eurosat.py","file_url":"https://github.com/yuanjiesh/rc3p/blob/HEAD/models/resnet_eurosat.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2b5240bcfd4cf3a"}},{"code_sha256_prefix":"607bb2c5904df9e5","entry":"resnet18","repo":"yuanjiesh/rc3p","repo_kind":"official","path":"models/resnet_tiny.py","file_url":"https://github.com/yuanjiesh/rc3p/blob/HEAD/models/resnet_tiny.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"607bb2c5904df9e5"}},{"code_sha256_prefix":"682ea0b23a9b9564","entry":"resnet20","repo":"yuanjiesh/rc3p","repo_kind":"official","path":"models/resnet_cifar.py","file_url":"https://github.com/yuanjiesh/rc3p/blob/HEAD/models/resnet_cifar.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"682ea0b23a9b9564"}},{"code_sha256_prefix":"f332591908476ccc","entry":"resnet20","repo":"yuanjiesh/rc3p","repo_kind":"official","path":"models/resnet_eurosat.py","file_url":"https://github.com/yuanjiesh/rc3p/blob/HEAD/models/resnet_eurosat.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f332591908476ccc"}},{"code_sha256_prefix":"c855e300f1337b9f","entry":"resnet32","repo":"yuanjiesh/rc3p","repo_kind":"official","path":"models/resnet_cifar.py","file_url":"https://github.com/yuanjiesh/rc3p/blob/HEAD/models/resnet_cifar.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c855e300f1337b9f"}},{"code_sha256_prefix":"11d038f53d77a8d2","entry":"resnet32","repo":"yuanjiesh/rc3p","repo_kind":"official","path":"models/resnet_eurosat.py","file_url":"https://github.com/yuanjiesh/rc3p/blob/HEAD/models/resnet_eurosat.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"11d038f53d77a8d2"}},{"code_sha256_prefix":"06958241b55ab4aa","entry":"resnet44","repo":"yuanjiesh/rc3p","repo_kind":"official","path":"models/resnet_cifar.py","file_url":"https://github.com/yuanjiesh/rc3p/blob/HEAD/models/resnet_cifar.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"06958241b55ab4aa"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}