{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/no-true-state-of-the-art-ood-detection","title":"No True State-of-the-Art? OOD Detection Methods are Inconsistent across Datasets","arxiv_id":"2109.05554","date":"2021-09-12","proceeding":null,"authors":["Fahim Tajwar","Ananya Kumar","Sang Michael Xie","Percy Liang"],"abstract":"Out-of-distribution detection is an important component of reliable ML systems. Prior literature has proposed various methods (e.g., MSP (Hendrycks & Gimpel, 2017), ODIN (Liang et al., 2018), Mahalanobis (Lee et al., 2018)), claiming they are state-of-the-art by showing they outperform previous methods on a selected set of in-distribution (ID) and out-of-distribution (OOD) datasets. In this work, we show that none of these methods are inherently better at OOD detection than others on a standardized set of 16 (ID, OOD) pairs. We give possible explanations for these inconsistencies with simple toy datasets where whether one method outperforms another depends on the structure of the ID and OOD datasets in question. Finally, we show that a method outperforming another on a certain (ID, OOD) pair may not do so in a low-data regime. In the low-data regime, we propose a distance-based method, Pairwise OOD detection (POD), which is based on Siamese networks and improves over Mahalanobis by sidestepping the expensive covariance estimation step. Our results suggest that the OOD detection problem may be too broad, and we should consider more specific structures for leverage.","url_abs":"https://arxiv.org/abs/2109.05554v1","url_pdf":"https://arxiv.org/pdf/2109.05554v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"no-true-state-of-the-art-ood-detection","repo_url":"https://github.com/tajwarfahim/ood_detection_inconsistency","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"ood-detection","task_name":"Out of Distribution (OOD) Detection"},{"task_slug":"out-of-distribution-detection","task_name":"Out-of-Distribution Detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2109.05554","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2109.05554"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tajwarfahim/ood_detection_inconsistency","reach":null}],"summary":{"ran":7,"ran_draft_wrong":4,"unverified":14},"by_repo_kind":{"official":{"samples":25,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":25,"samples":[{"code_sha256_prefix":"a04706445c3a44b5","entry":"LinearCritic","repo":"tajwarfahim/ood_detection_inconsistency","repo_kind":"official","path":"utils/siamese_network.py","file_url":"https://github.com/tajwarfahim/ood_detection_inconsistency/blob/HEAD/utils/siamese_network.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a04706445c3a44b5"}},{"code_sha256_prefix":"0f5c726e9580ca0a","entry":"ResNet","repo":"tajwarfahim/ood_detection_inconsistency","repo_kind":"official","path":"utils/siamese_network.py","file_url":"https://github.com/tajwarfahim/ood_detection_inconsistency/blob/HEAD/utils/siamese_network.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0f5c726e9580ca0a"}},{"code_sha256_prefix":"7ca36149b988c54e","entry":"ResNet18","repo":"tajwarfahim/ood_detection_inconsistency","repo_kind":"official","path":"utils/siamese_network.py","file_url":"https://github.com/tajwarfahim/ood_detection_inconsistency/blob/HEAD/utils/siamese_network.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7ca36149b988c54e"}},{"code_sha256_prefix":"9d4e412922385a20","entry":"ResNet34","repo":"tajwarfahim/ood_detection_inconsistency","repo_kind":"official","path":"utils/siamese_network.py","file_url":"https://github.com/tajwarfahim/ood_detection_inconsistency/blob/HEAD/utils/siamese_network.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9d4e412922385a20"}},{"code_sha256_prefix":"caa63b6cbc7fd52e","entry":"ResNet50","repo":"tajwarfahim/ood_detection_inconsistency","repo_kind":"official","path":"utils/siamese_network.py","file_url":"https://github.com/tajwarfahim/ood_detection_inconsistency/blob/HEAD/utils/siamese_network.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"caa63b6cbc7fd52e"}},{"code_sha256_prefix":"a1333c964477658d","entry":"ResNetWithoutLastLayer","repo":"tajwarfahim/ood_detection_inconsistency","repo_kind":"official","path":"utils/siamese_network.py","file_url":"https://github.com/tajwarfahim/ood_detection_inconsistency/blob/HEAD/utils/siamese_network.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a1333c964477658d"}},{"code_sha256_prefix":"ef84d9b3e36ef6bb","entry":"SiameseNetworkVersion1","repo":"tajwarfahim/ood_detection_inconsistency","repo_kind":"official","path":"utils/siamese_network.py","file_url":"https://github.com/tajwarfahim/ood_detection_inconsistency/blob/HEAD/utils/siamese_network.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ef84d9b3e36ef6bb"}},{"code_sha256_prefix":"091727a59cfe18c5","entry":"SiameseNetworkVersion2","repo":"tajwarfahim/ood_detection_inconsistency","repo_kind":"official","path":"utils/siamese_network.py","file_url":"https://github.com/tajwarfahim/ood_detection_inconsistency/blob/HEAD/utils/siamese_network.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"091727a59cfe18c5"}},{"code_sha256_prefix":"bb726f48afa2bf1e","entry":"SiameseNetworkVersion3","repo":"tajwarfahim/ood_detection_inconsistency","repo_kind":"official","path":"utils/siamese_network.py","file_url":"https://github.com/tajwarfahim/ood_detection_inconsistency/blob/HEAD/utils/siamese_network.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bb726f48afa2bf1e"}},{"code_sha256_prefix":"b655b80a757b91a5","entry":"SiameseNetworkVersion5","repo":"tajwarfahim/ood_detection_inconsistency","repo_kind":"official","path":"utils/siamese_network.py","file_url":"https://github.com/tajwarfahim/ood_detection_inconsistency/blob/HEAD/utils/siamese_network.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b655b80a757b91a5"}},{"code_sha256_prefix":"a8193ab021196916","entry":"process_shared_model_name_wide_resnet","repo":"tajwarfahim/ood_detection_inconsistency","repo_kind":"official","path":"utils/siamese_network.py","file_url":"https://github.com/tajwarfahim/ood_detection_inconsistency/blob/HEAD/utils/siamese_network.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a8193ab021196916"}},{"code_sha256_prefix":"e113c4f6c24ebbd1","entry":"SiameseNetworkSkeleton","repo":"tajwarfahim/ood_detection_inconsistency","repo_kind":"official","path":"utils/siamese_network.py","file_url":"https://github.com/tajwarfahim/ood_detection_inconsistency/blob/HEAD/utils/siamese_network.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e113c4f6c24ebbd1"}},{"code_sha256_prefix":"0b7f4ef94c512dfd","entry":"SiameseNetworkVersion4","repo":"tajwarfahim/ood_detection_inconsistency","repo_kind":"official","path":"utils/siamese_network.py","file_url":"https://github.com/tajwarfahim/ood_detection_inconsistency/blob/HEAD/utils/siamese_network.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0b7f4ef94c512dfd"}},{"code_sha256_prefix":"e005fe22e69cd426","entry":"WideResNet","repo":"tajwarfahim/ood_detection_inconsistency","repo_kind":"official","path":"utils/siamese_network.py","file_url":"https://github.com/tajwarfahim/ood_detection_inconsistency/blob/HEAD/utils/siamese_network.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e005fe22e69cd426"}},{"code_sha256_prefix":"c175203550f4dcb0","entry":"WideResNetWithoutLastLayer","repo":"tajwarfahim/ood_detection_inconsistency","repo_kind":"official","path":"utils/siamese_network.py","file_url":"https://github.com/tajwarfahim/ood_detection_inconsistency/blob/HEAD/utils/siamese_network.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c175203550f4dcb0"}},{"code_sha256_prefix":"84ab954d28da2dfd","entry":"createSiameseNetwork","repo":"tajwarfahim/ood_detection_inconsistency","repo_kind":"official","path":"utils/siamese_network.py","file_url":"https://github.com/tajwarfahim/ood_detection_inconsistency/blob/HEAD/utils/siamese_network.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"84ab954d28da2dfd"}},{"code_sha256_prefix":"31f08a3d3a6f84f3","entry":"create_resnet","repo":"tajwarfahim/ood_detection_inconsistency","repo_kind":"official","path":"utils/siamese_network.py","file_url":"https://github.com/tajwarfahim/ood_detection_inconsistency/blob/HEAD/utils/siamese_network.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"31f08a3d3a6f84f3"}},{"code_sha256_prefix":"e893a80255c1f421","entry":"create_resnet","repo":"tajwarfahim/ood_detection_inconsistency","repo_kind":"official","path":"utils/siamese_network.py","file_url":"https://github.com/tajwarfahim/ood_detection_inconsistency/blob/HEAD/utils/siamese_network.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e893a80255c1f421"}},{"code_sha256_prefix":"1bf44f5fcc3432c6","entry":"create_resnet_based_shared_model","repo":"tajwarfahim/ood_detection_inconsistency","repo_kind":"official","path":"utils/siamese_network.py","file_url":"https://github.com/tajwarfahim/ood_detection_inconsistency/blob/HEAD/utils/siamese_network.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1bf44f5fcc3432c6"}},{"code_sha256_prefix":"1655bbb2f181e73d","entry":"create_siamese_network_wrapper","repo":"tajwarfahim/ood_detection_inconsistency","repo_kind":"official","path":"utils/siamese_network.py","file_url":"https://github.com/tajwarfahim/ood_detection_inconsistency/blob/HEAD/utils/siamese_network.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1655bbb2f181e73d"}},{"code_sha256_prefix":"cfd72ce7f48cf580","entry":"create_wide_resnet","repo":"tajwarfahim/ood_detection_inconsistency","repo_kind":"official","path":"utils/siamese_network.py","file_url":"https://github.com/tajwarfahim/ood_detection_inconsistency/blob/HEAD/utils/siamese_network.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cfd72ce7f48cf580"}},{"code_sha256_prefix":"3d3f74f6425de128","entry":"create_wide_resnet_based_shared_model","repo":"tajwarfahim/ood_detection_inconsistency","repo_kind":"official","path":"utils/siamese_network.py","file_url":"https://github.com/tajwarfahim/ood_detection_inconsistency/blob/HEAD/utils/siamese_network.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3d3f74f6425de128"}},{"code_sha256_prefix":"adefa2d05ffc3cb9","entry":"create_wide_resnet_based_shared_model","repo":"tajwarfahim/ood_detection_inconsistency","repo_kind":"official","path":"utils/siamese_network.py","file_url":"https://github.com/tajwarfahim/ood_detection_inconsistency/blob/HEAD/utils/siamese_network.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"adefa2d05ffc3cb9"}},{"code_sha256_prefix":"623dbfda864bfef7","entry":"print_message","repo":"tajwarfahim/ood_detection_inconsistency","repo_kind":"official","path":"utils/siamese_network.py","file_url":"https://github.com/tajwarfahim/ood_detection_inconsistency/blob/HEAD/utils/siamese_network.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"623dbfda864bfef7"}},{"code_sha256_prefix":"39a439e6c10d840b","entry":"process_shared_model_name_wide_resnet","repo":"tajwarfahim/ood_detection_inconsistency","repo_kind":"official","path":"utils/siamese_network.py","file_url":"https://github.com/tajwarfahim/ood_detection_inconsistency/blob/HEAD/utils/siamese_network.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39a439e6c10d840b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}