{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/does-your-model-know-the-digit-6-is-not-a-cat","title":"A Less Biased Evaluation of Out-of-distribution Sample Detectors","arxiv_id":"1809.04729","date":"2018-09-13","proceeding":null,"authors":["Alireza Shafaei","Mark Schmidt","James J. Little"],"abstract":"In the real world, a learning system could receive an input that is unlike anything it has seen during training. Unfortunately, out-of-distribution samples can lead to unpredictable behaviour. We need to know whether any given input belongs to the population distribution of the training/evaluation data to prevent unpredictable behaviour in deployed systems. A recent surge of interest in this problem has led to the development of sophisticated techniques in the deep learning literature. However, due to the absence of a standard problem definition or an exhaustive evaluation, it is not evident if we can rely on these methods. What makes this problem different from a typical supervised learning setting is that the distribution of outliers used in training may not be the same as the distribution of outliers encountered in the application. Classical approaches that learn inliers vs. outliers with only two datasets can yield optimistic results. We introduce OD-test, a three-dataset evaluation scheme as a more reliable strategy to assess progress on this problem. We present an exhaustive evaluation of a broad set of methods from related areas on image classification tasks. Contrary to the existing results, we show that for realistic applications of high-dimensional images the previous techniques have low accuracy and are not reliable in practice.","url_abs":"https://arxiv.org/abs/1809.04729v2","url_pdf":"https://arxiv.org/pdf/1809.04729v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"does-your-model-know-the-digit-6-is-not-a-cat","repo_url":"https://github.com/ashafaei/OD-test","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"does-your-model-know-the-digit-6-is-not-a-cat","repo_url":"https://github.com/caotians1/OD-test-master","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"does-your-model-know-the-digit-6-is-not-a-cat","repo_url":"https://github.com/safednn-group/nap-ood","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1809.04729","atlas_url":"https://app.syntology.ai/?focus=1809.04729","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1809.04729"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/safednn-group/nap-ood","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/caotians1/OD-test-master","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ashafaei/OD-test","reach":null}],"summary":{"ran":2,"ran_violates":1,"unverified":2},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"52d10a10fb72fb9f","entry":"find_classes","repo":"safednn-group/nap-ood","repo_kind":"listed","path":"datasets/TinyImagenet.py","file_url":"https://github.com/safednn-group/nap-ood/blob/HEAD/datasets/TinyImagenet.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"52d10a10fb72fb9f"}},{"code_sha256_prefix":"276cf6be1608642f","entry":"has_done_before","repo":"ashafaei/OD-test","repo_kind":"official","path":"eval3d.py","file_url":"https://github.com/ashafaei/OD-test/blob/HEAD/eval3d.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"276cf6be1608642f"}},{"code_sha256_prefix":"4e6a8faaf8e44abe","entry":"is_image_file","repo":"safednn-group/nap-ood","repo_kind":"listed","path":"datasets/TinyImagenet.py","file_url":"https://github.com/safednn-group/nap-ood/blob/HEAD/datasets/TinyImagenet.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4e6a8faaf8e44abe"}},{"code_sha256_prefix":"ff620eb5222a4395","entry":"has_done_before","repo":"safednn-group/nap-ood","repo_kind":"listed","path":"eval3d.py","file_url":"https://github.com/safednn-group/nap-ood/blob/HEAD/eval3d.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ff620eb5222a4395"}},{"code_sha256_prefix":"59962a74509c9b28","entry":"make_dataset","repo":"safednn-group/nap-ood","repo_kind":"listed","path":"datasets/TinyImagenet.py","file_url":"https://github.com/safednn-group/nap-ood/blob/HEAD/datasets/TinyImagenet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"59962a74509c9b28"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}