{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/counterfactually-comparing-abstaining-1","title":"Counterfactually Comparing Abstaining Classifiers","arxiv_id":"2305.10564","date":"2023-05-17","proceeding":"NeurIPS 2023 11","authors":["Yo Joong Choe","Aditya Gangrade","Aaditya Ramdas"],"abstract":"Abstaining classifiers have the option to abstain from making predictions on inputs that they are unsure about. These classifiers are becoming increasingly popular in high-stakes decision-making problems, as they can withhold uncertain predictions to improve their reliability and safety. When evaluating black-box abstaining classifier(s), however, we lack a principled approach that accounts for what the classifier would have predicted on its abstentions. These missing predictions matter when they can eventually be utilized, either directly or as a backup option in a failure mode. In this paper, we introduce a novel approach and perspective to the problem of evaluating and comparing abstaining classifiers by treating abstentions as missing data. Our evaluation approach is centered around defining the counterfactual score of an abstaining classifier, defined as the expected performance of the classifier had it not been allowed to abstain. We specify the conditions under which the counterfactual score is identifiable: if the abstentions are stochastic, and if the evaluation data is independent of the training data (ensuring that the predictions are missing at random), then the score is identifiable. Note that, if abstentions are deterministic, then the score is unidentifiable because the classifier can perform arbitrarily poorly on its abstentions. Leveraging tools from observational causal inference, we then develop nonparametric and doubly robust methods to efficiently estimate this quantity under identification. Our approach is examined in both simulated and real data experiments.","url_abs":"https://arxiv.org/abs/2305.10564v2","url_pdf":"https://arxiv.org/pdf/2305.10564v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"counterfactually-comparing-abstaining-1","repo_url":"https://github.com/yjchoe/comparingabstainingclassifiers","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"causal-inference","task_name":"Causal Inference"},{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":null,"task_name":"counterfactual"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.10564","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.10564"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/yjchoe/ComparingAbstainingClassifiers","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yjchoe/comparingabstainingclassifiers","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":9},"by_repo_kind":{"official":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7dc1be337fe9d277","entry":"compute_confidence","repo":"yjchoe/ComparingAbstainingClassifiers","repo_kind":"official","path":"comparecast_causal/abstaining_predictors.py","file_url":"https://github.com/yjchoe/ComparingAbstainingClassifiers/blob/HEAD/comparecast_causal/abstaining_predictors.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7dc1be337fe9d277"}},{"code_sha256_prefix":"674c094809544cb9","entry":"confint_clt","repo":"yjchoe/ComparingAbstainingClassifiers","repo_kind":"official","path":"comparecast_causal/drconfint.py","file_url":"https://github.com/yjchoe/ComparingAbstainingClassifiers/blob/HEAD/comparecast_causal/drconfint.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"674c094809544cb9"}},{"code_sha256_prefix":"d24633be058fec72","entry":"convert_to_onehot","repo":"yjchoe/ComparingAbstainingClassifiers","repo_kind":"official","path":"comparecast_causal/utils.py","file_url":"https://github.com/yjchoe/ComparingAbstainingClassifiers/blob/HEAD/comparecast_causal/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d24633be058fec72"}},{"code_sha256_prefix":"1e02678d06f7e01c","entry":"convert_to_ordinal","repo":"yjchoe/ComparingAbstainingClassifiers","repo_kind":"official","path":"comparecast_causal/utils.py","file_url":"https://github.com/yjchoe/ComparingAbstainingClassifiers/blob/HEAD/comparecast_causal/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1e02678d06f7e01c"}},{"code_sha256_prefix":"f1f5e0682bdb9c8f","entry":"generate_binary_mar","repo":"yjchoe/ComparingAbstainingClassifiers","repo_kind":"official","path":"comparecast_causal/data_utils/abstaining_classifiers.py","file_url":"https://github.com/yjchoe/ComparingAbstainingClassifiers/blob/HEAD/comparecast_causal/data_utils/abstaining_classifiers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1f5e0682bdb9c8f"}},{"code_sha256_prefix":"e3b49a725b942885","entry":"generate_differently_imbalanced_data","repo":"yjchoe/ComparingAbstainingClassifiers","repo_kind":"official","path":"comparecast_causal/data_utils/abstaining_classifiers.py","file_url":"https://github.com/yjchoe/ComparingAbstainingClassifiers/blob/HEAD/comparecast_causal/data_utils/abstaining_classifiers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e3b49a725b942885"}},{"code_sha256_prefix":"4411536d6d628e67","entry":"make_linear_classifier","repo":"yjchoe/ComparingAbstainingClassifiers","repo_kind":"official","path":"comparecast_causal/learners.py","file_url":"https://github.com/yjchoe/ComparingAbstainingClassifiers/blob/HEAD/comparecast_causal/learners.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4411536d6d628e67"}},{"code_sha256_prefix":"03179e90b0c25205","entry":"make_linear_regressor","repo":"yjchoe/ComparingAbstainingClassifiers","repo_kind":"official","path":"comparecast_causal/learners.py","file_url":"https://github.com/yjchoe/ComparingAbstainingClassifiers/blob/HEAD/comparecast_causal/learners.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"03179e90b0c25205"}},{"code_sha256_prefix":"b57a4f5daec383d9","entry":"predict_or_abstain","repo":"yjchoe/ComparingAbstainingClassifiers","repo_kind":"official","path":"comparecast_causal/abstaining_predictors.py","file_url":"https://github.com/yjchoe/ComparingAbstainingClassifiers/blob/HEAD/comparecast_causal/abstaining_predictors.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b57a4f5daec383d9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}