{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improving-selective-classification","title":"How to Fix a Broken Confidence Estimator: Evaluating Post-hoc Methods for Selective Classification with Deep Neural Networks","arxiv_id":"2305.15508","date":"2023-05-24","proceeding":null,"authors":["Luís Felipe P. Cattelan","Danilo Silva"],"abstract":"This paper addresses the problem of selective classification for deep neural networks, where a model is allowed to abstain from low-confidence predictions to avoid potential errors. We focus on so-called post-hoc methods, which replace the confidence estimator of a given classifier without modifying or retraining it, thus being practically appealing. Considering neural networks with softmax outputs, our goal is to identify the best confidence estimator that can be computed directly from the unnormalized logits. This problem is motivated by the intriguing observation in recent work that many classifiers appear to have a \"broken\" confidence estimator, in the sense that their selective classification performance is much worse than what could be expected by their corresponding accuracies. We perform an extensive experimental study of many existing and proposed confidence estimators applied to 84 pretrained ImageNet classifiers available from popular repositories. Our results show that a simple $p$-norm normalization of the logits, followed by taking the maximum logit as the confidence estimator, can lead to considerable gains in selective classification performance, completely fixing the pathological behavior observed in many classifiers. As a consequence, the selective classification performance of any classifier becomes almost entirely determined by its corresponding accuracy. Moreover, these results are shown to be consistent under distribution shift. Our code is available at https://github.com/lfpc/FixSelectiveClassification.","url_abs":"https://arxiv.org/abs/2305.15508v4","url_pdf":"https://arxiv.org/pdf/2305.15508v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improving-selective-classification","repo_url":"https://github.com/lfpc/fixselectiveclassification","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"classification-1","task_name":"Classification"}],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.15508","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.15508"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lfpc/fixselectiveclassification","reach":null}],"summary":{"ran_honours":3,"ran_fixture":2,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b12a91154b8850bc","entry":"MSP","repo":"lfpc/fixselectiveclassification","repo_kind":"official","path":"utils/measures.py","file_url":"https://github.com/lfpc/fixselectiveclassification/blob/HEAD/utils/measures.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b12a91154b8850bc"}},{"code_sha256_prefix":"195700cc198edb8c","entry":"RC_curve","repo":"lfpc/fixselectiveclassification","repo_kind":"official","path":"utils/metrics.py","file_url":"https://github.com/lfpc/fixselectiveclassification/blob/HEAD/utils/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"195700cc198edb8c"}},{"code_sha256_prefix":"de643ae43b93bf74","entry":"ROC_curve","repo":"lfpc/fixselectiveclassification","repo_kind":"official","path":"utils/metrics.py","file_url":"https://github.com/lfpc/fixselectiveclassification/blob/HEAD/utils/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"de643ae43b93bf74"}},{"code_sha256_prefix":"1c1ca7bac1d99a0a","entry":"entropy","repo":"lfpc/fixselectiveclassification","repo_kind":"official","path":"utils/measures.py","file_url":"https://github.com/lfpc/fixselectiveclassification/blob/HEAD/utils/measures.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1c1ca7bac1d99a0a"}},{"code_sha256_prefix":"74c0813262a3764f","entry":"negative_entropy","repo":"lfpc/fixselectiveclassification","repo_kind":"official","path":"utils/measures.py","file_url":"https://github.com/lfpc/fixselectiveclassification/blob/HEAD/utils/measures.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"74c0813262a3764f"}},{"code_sha256_prefix":"6c4c7a19a1e55274","entry":"accuracy","repo":"lfpc/fixselectiveclassification","repo_kind":"official","path":"utils/metrics.py","file_url":"https://github.com/lfpc/fixselectiveclassification/blob/HEAD/utils/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6c4c7a19a1e55274"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}