{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deep-neural-network-benchmarks-for-selective","title":"Deep Neural Network Benchmarks for Selective Classification","arxiv_id":"2401.12708","date":"2024-01-23","proceeding":null,"authors":["Andrea Pugnana","Lorenzo Perini","Jesse Davis","Salvatore Ruggieri"],"abstract":"With the increasing deployment of machine learning models in many socially sensitive tasks, there is a growing demand for reliable and trustworthy predictions. One way to accomplish these requirements is to allow a model to abstain from making a prediction when there is a high risk of making an error. This requires adding a selection mechanism to the model, which selects those examples for which the model will provide a prediction. The selective classification framework aims to design a mechanism that balances the fraction of rejected predictions (i.e., the proportion of examples for which the model does not make a prediction) versus the improvement in predictive performance on the selected predictions. Multiple selective classification frameworks exist, most of which rely on deep neural network architectures. However, the empirical evaluation of the existing approaches is still limited to partial comparisons among methods and settings, providing practitioners with little insight into their relative merits. We fill this gap by benchmarking 18 baselines on a diverse set of 44 datasets that includes both image and tabular data. Moreover, there is a mix of binary and multiclass tasks. We evaluate these approaches using several criteria, including selective error rate, empirical coverage, distribution of rejected instance's classes, and performance on out-of-distribution instances. The results indicate that there is not a single clear winner among the surveyed baselines, and the best method depends on the users' objectives.","url_abs":"https://arxiv.org/abs/2401.12708v2","url_pdf":"https://arxiv.org/pdf/2401.12708v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deep-neural-network-benchmarks-for-selective","repo_url":"https://github.com/andrepugni/esc","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"classification-1","task_name":"Classification"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2401.12708","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.12708"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/andrepugni/esc","reach":{"status":"ok"}}],"summary":{"ran":9,"unverified":6},"by_repo_kind":{"official":{"samples":15,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"a2e88427192d44ee","entry":"auc_mu","repo":"andrepugni/esc","repo_kind":"official","path":"classes/losses.py","file_url":"https://github.com/andrepugni/esc/blob/HEAD/classes/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a2e88427192d44ee"}},{"code_sha256_prefix":"ba7123cce304c1cf","entry":"cast","repo":"andrepugni/esc","repo_kind":"official","path":"classes/modules.py","file_url":"https://github.com/andrepugni/esc/blob/HEAD/classes/modules.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ba7123cce304c1cf"}},{"code_sha256_prefix":"de73b78fe4c2e0fb","entry":"check_architecture","repo":"andrepugni/esc","repo_kind":"official","path":"testing_ood.py","file_url":"https://github.com/andrepugni/esc/blob/HEAD/testing_ood.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"de73b78fe4c2e0fb"}},{"code_sha256_prefix":"e33526662b5a5e89","entry":"cross_entropy_selection_vectorized","repo":"andrepugni/esc","repo_kind":"official","path":"classes/losses.py","file_url":"https://github.com/andrepugni/esc/blob/HEAD/classes/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e33526662b5a5e89"}},{"code_sha256_prefix":"60390786584e0e87","entry":"cross_entropy_vectorized","repo":"andrepugni/esc","repo_kind":"official","path":"classes/losses.py","file_url":"https://github.com/andrepugni/esc/blob/HEAD/classes/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"60390786584e0e87"}},{"code_sha256_prefix":"1d0da351b9ca00b4","entry":"disable_dropout","repo":"andrepugni/esc","repo_kind":"official","path":"classes/utils.py","file_url":"https://github.com/andrepugni/esc/blob/HEAD/classes/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1d0da351b9ca00b4"}},{"code_sha256_prefix":"0d78ebda880af125","entry":"normalization","repo":"andrepugni/esc","repo_kind":"official","path":"classes/datasets.py","file_url":"https://github.com/andrepugni/esc/blob/HEAD/classes/datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0d78ebda880af125"}},{"code_sha256_prefix":"e58f49dd7ea65e34","entry":"onlytransformation","repo":"andrepugni/esc","repo_kind":"official","path":"classes/datasets.py","file_url":"https://github.com/andrepugni/esc/blob/HEAD/classes/datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e58f49dd7ea65e34"}},{"code_sha256_prefix":"dc5c6cf51387278d","entry":"transformation","repo":"andrepugni/esc","repo_kind":"official","path":"classes/datasets.py","file_url":"https://github.com/andrepugni/esc/blob/HEAD/classes/datasets.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dc5c6cf51387278d"}},{"code_sha256_prefix":"8e5dd28151793170","entry":"buildSelNet","repo":"andrepugni/esc","repo_kind":"official","path":"classes/utils.py","file_url":"https://github.com/andrepugni/esc/blob/HEAD/classes/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8e5dd28151793170"}},{"code_sha256_prefix":"e60ec9043c30497f","entry":"calibrate_aucross","repo":"andrepugni/esc","repo_kind":"official","path":"testing_max_coverage.py","file_url":"https://github.com/andrepugni/esc/blob/HEAD/testing_max_coverage.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e60ec9043c30497f"}},{"code_sha256_prefix":"5e3c288c420ffa78","entry":"calibrate_aucross","repo":"andrepugni/esc","repo_kind":"official","path":"testing_ood.py","file_url":"https://github.com/andrepugni/esc/blob/HEAD/testing_ood.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5e3c288c420ffa78"}},{"code_sha256_prefix":"2709d19c471a0f05","entry":"load_data","repo":"andrepugni/esc","repo_kind":"official","path":"testing.py","file_url":"https://github.com/andrepugni/esc/blob/HEAD/testing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2709d19c471a0f05"}},{"code_sha256_prefix":"64eb1db87c2e5493","entry":"make_layers","repo":"andrepugni/esc","repo_kind":"official","path":"classes/utils.py","file_url":"https://github.com/andrepugni/esc/blob/HEAD/classes/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"64eb1db87c2e5493"}},{"code_sha256_prefix":"acfc9110c70967de","entry":"make_optimizer","repo":"andrepugni/esc","repo_kind":"official","path":"testing_ood.py","file_url":"https://github.com/andrepugni/esc/blob/HEAD/testing_ood.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"acfc9110c70967de"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}