{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-unified-benchmark-for-the-unknown-detection","title":"A Unified Benchmark for the Unknown Detection Capability of Deep Neural Networks","arxiv_id":"2112.00337","date":"2021-12-01","proceeding":null,"authors":["Jihyo Kim","Jiin Koo","Sangheum Hwang"],"abstract":"Deep neural networks have achieved outstanding performance over various tasks, but they have a critical issue: over-confident predictions even for completely unknown samples. Many studies have been proposed to successfully filter out these unknown samples, but they only considered narrow and specific tasks, referred to as misclassification detection, open-set recognition, or out-of-distribution detection. In this work, we argue that these tasks should be treated as fundamentally an identical problem because an ideal model should possess detection capability for all those tasks. Therefore, we introduce the unknown detection task, an integration of previous individual tasks, for a rigorous examination of the detection capability of deep neural networks on a wide spectrum of unknown samples. To this end, unified benchmark datasets on different scales were constructed and the unknown detection capabilities of existing popular methods were subject to comparison. We found that Deep Ensemble consistently outperforms the other approaches in detecting unknowns; however, all methods are only successful for a specific type of unknown. The reproducible code and benchmark datasets are available at https://github.com/daintlab/unknown-detection-benchmarks .","url_abs":"https://arxiv.org/abs/2112.00337v2","url_pdf":"https://arxiv.org/pdf/2112.00337v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-unified-benchmark-for-the-unknown-detection","repo_url":"https://github.com/daintlab/unknown-detection-benchmarks","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"open-set-learning","task_name":"Open Set Learning"},{"task_slug":"out-of-distribution-detection","task_name":"Out-of-Distribution Detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2112.00337","atlas_url":"https://app.syntology.ai/?focus=2112.00337","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2112.00337"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/daintlab/unknown-detection-benchmarks","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"ran":7,"ran_draft_wrong":3,"ran_honours":2,"ran_violates":1,"unverified":1},"by_repo_kind":{"official":{"samples":15,"ran":14,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"67a13c0cb19afb83","entry":"accuracy","repo":"daintlab/unknown-detection-benchmarks","repo_kind":"official","path":"utils.py","file_url":"https://github.com/daintlab/unknown-detection-benchmarks/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"67a13c0cb19afb83"}},{"code_sha256_prefix":"612bae70040c203f","entry":"aurc_eaurc","repo":"daintlab/unknown-detection-benchmarks","repo_kind":"official","path":"metrics_md.py","file_url":"https://github.com/daintlab/unknown-detection-benchmarks/blob/HEAD/metrics_md.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"612bae70040c203f"}},{"code_sha256_prefix":"d9def42110729a85","entry":"conv1x1","repo":"daintlab/unknown-detection-benchmarks","repo_kind":"official","path":"models/resnet_imagenet.py","file_url":"https://github.com/daintlab/unknown-detection-benchmarks/blob/HEAD/models/resnet_imagenet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d9def42110729a85"}},{"code_sha256_prefix":"160bb14bd76201b4","entry":"conv3x3","repo":"daintlab/unknown-detection-benchmarks","repo_kind":"official","path":"models/resnet_imagenet.py","file_url":"https://github.com/daintlab/unknown-detection-benchmarks/blob/HEAD/models/resnet_imagenet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"160bb14bd76201b4"}},{"code_sha256_prefix":"6af95ebe99af2e36","entry":"conv3x3","repo":"daintlab/unknown-detection-benchmarks","repo_kind":"official","path":"models/resnet_cifar.py","file_url":"https://github.com/daintlab/unknown-detection-benchmarks/blob/HEAD/models/resnet_cifar.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6af95ebe99af2e36"}},{"code_sha256_prefix":"be1cf2403304fa6d","entry":"float_parameter","repo":"daintlab/unknown-detection-benchmarks","repo_kind":"official","path":"methods/augmix/augmentations.py","file_url":"https://github.com/daintlab/unknown-detection-benchmarks/blob/HEAD/methods/augmix/augmentations.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"be1cf2403304fa6d"}},{"code_sha256_prefix":"ed35359a5c2b419c","entry":"get_curve","repo":"daintlab/unknown-detection-benchmarks","repo_kind":"official","path":"metrics_ood.py","file_url":"https://github.com/daintlab/unknown-detection-benchmarks/blob/HEAD/metrics_ood.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed35359a5c2b419c"}},{"code_sha256_prefix":"1693518f0ab9445d","entry":"in_dist_loader","repo":"daintlab/unknown-detection-benchmarks","repo_kind":"official","path":"dataloader.py","file_url":"https://github.com/daintlab/unknown-detection-benchmarks/blob/HEAD/dataloader.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1693518f0ab9445d"}},{"code_sha256_prefix":"beaa91443124b324","entry":"int_parameter","repo":"daintlab/unknown-detection-benchmarks","repo_kind":"official","path":"methods/augmix/augmentations.py","file_url":"https://github.com/daintlab/unknown-detection-benchmarks/blob/HEAD/methods/augmix/augmentations.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"beaa91443124b324"}},{"code_sha256_prefix":"d018404cd06ecc3f","entry":"one_hot_embedding","repo":"daintlab/unknown-detection-benchmarks","repo_kind":"official","path":"utils.py","file_url":"https://github.com/daintlab/unknown-detection-benchmarks/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d018404cd06ecc3f"}},{"code_sha256_prefix":"5cc8d4764ac07a35","entry":"sample_level","repo":"daintlab/unknown-detection-benchmarks","repo_kind":"official","path":"methods/augmix/augmentations.py","file_url":"https://github.com/daintlab/unknown-detection-benchmarks/blob/HEAD/methods/augmix/augmentations.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5cc8d4764ac07a35"}},{"code_sha256_prefix":"9052013b7e166b05","entry":"str2bool","repo":"daintlab/unknown-detection-benchmarks","repo_kind":"official","path":"utils.py","file_url":"https://github.com/daintlab/unknown-detection-benchmarks/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9052013b7e166b05"}},{"code_sha256_prefix":"10d0fca58f787cd4","entry":"trn_loader","repo":"daintlab/unknown-detection-benchmarks","repo_kind":"official","path":"dataloader.py","file_url":"https://github.com/daintlab/unknown-detection-benchmarks/blob/HEAD/dataloader.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"10d0fca58f787cd4"}},{"code_sha256_prefix":"d3690f3ae6c59a20","entry":"tst_loader","repo":"daintlab/unknown-detection-benchmarks","repo_kind":"official","path":"dataloader.py","file_url":"https://github.com/daintlab/unknown-detection-benchmarks/blob/HEAD/dataloader.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d3690f3ae6c59a20"}},{"code_sha256_prefix":"f1558cd7a9612567","entry":"resnet18","repo":"daintlab/unknown-detection-benchmarks","repo_kind":"official","path":"models/resnet_imagenet.py","file_url":"https://github.com/daintlab/unknown-detection-benchmarks/blob/HEAD/models/resnet_imagenet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1558cd7a9612567"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}