{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/openood-benchmarking-generalized-out-of","title":"OpenOOD: Benchmarking Generalized Out-of-Distribution Detection","arxiv_id":"2210.07242","date":"2022-10-13","proceeding":null,"authors":["Jingkang Yang","Pengyun Wang","Dejian Zou","Zitang Zhou","Kunyuan Ding","Wenxuan Peng","Haoqi Wang","Guangyao Chen","Bo Li","Yiyou Sun","Xuefeng Du","Kaiyang Zhou","Wayne Zhang","Dan Hendrycks","Yixuan Li","Ziwei Liu"],"abstract":"Out-of-distribution (OOD) detection is vital to safety-critical machine learning applications and has thus been extensively studied, with a plethora of methods developed in the literature. However, the field currently lacks a unified, strictly formulated, and comprehensive benchmark, which often results in unfair comparisons and inconclusive results. From the problem setting perspective, OOD detection is closely related to neighboring fields including anomaly detection (AD), open set recognition (OSR), and model uncertainty, since methods developed for one domain are often applicable to each other. To help the community to improve the evaluation and advance, we build a unified, well-structured codebase called OpenOOD, which implements over 30 methods developed in relevant fields and provides a comprehensive benchmark under the recently proposed generalized OOD detection framework. With a comprehensive comparison of these methods, we are gratified that the field has progressed significantly over the past few years, where both preprocessing methods and the orthogonal post-hoc methods show strong potential.","url_abs":"https://arxiv.org/abs/2210.07242v1","url_pdf":"https://arxiv.org/pdf/2210.07242v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"openood-benchmarking-generalized-out-of","repo_url":"https://github.com/jingkang50/openood","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"openood-benchmarking-generalized-out-of","repo_url":"https://github.com/adverml/advopenood","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"openood-benchmarking-generalized-out-of","repo_url":"https://github.com/antoinedemathelin/openood","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"openood-benchmarking-generalized-out-of","repo_url":"https://github.com/stevenboys/moon","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"anomaly-detection","task_name":"Anomaly Detection"},{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"open-set-learning","task_name":"Open Set Learning"},{"task_slug":"ood-detection","task_name":"Out of Distribution (OOD) Detection"},{"task_slug":"out-of-distribution-detection","task_name":"Out-of-Distribution Detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.07242","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.07242"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Jingkang50/OpenOOD","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/stevenboys/moon","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jingkang50/openood","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/adverml/advopenood","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/antoinedemathelin/openood","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":5,"ran_honours":1,"ran_violates":2,"unverified":9},"by_repo_kind":{"listed":{"samples":17,"ran":8,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4c4dd64f6d883f76","entry":"check_file_ending","repo":"adverml/advopenood","repo_kind":"listed","path":"openood/attacks/misc.py","file_url":"https://github.com/adverml/advopenood/blob/HEAD/openood/attacks/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4c4dd64f6d883f76"}},{"code_sha256_prefix":"ee80cf467065076b","entry":"conv1x1","repo":"antoinedemathelin/openood","repo_kind":"listed","path":"openood/networks/bit.py","file_url":"https://github.com/antoinedemathelin/openood/blob/HEAD/openood/networks/bit.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ee80cf467065076b"}},{"code_sha256_prefix":"f01b8d3901f0b289","entry":"conv3x3","repo":"antoinedemathelin/openood","repo_kind":"listed","path":"openood/networks/bit.py","file_url":"https://github.com/antoinedemathelin/openood/blob/HEAD/openood/networks/bit.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f01b8d3901f0b289"}},{"code_sha256_prefix":"c86ea3831fc179f4","entry":"convert_to_float","repo":"adverml/advopenood","repo_kind":"listed","path":"openood/attacks/misc.py","file_url":"https://github.com/adverml/advopenood/blob/HEAD/openood/attacks/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c86ea3831fc179f4"}},{"code_sha256_prefix":"dbe36e80c469a4e2","entry":"load_args","repo":"adverml/advopenood","repo_kind":"listed","path":"openood/attacks/misc.py","file_url":"https://github.com/adverml/advopenood/blob/HEAD/openood/attacks/misc.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dbe36e80c469a4e2"}},{"code_sha256_prefix":"d8339d591bbd7f52","entry":"rew_ce","repo":"antoinedemathelin/openood","repo_kind":"listed","path":"openood/losses/reweight.py","file_url":"https://github.com/antoinedemathelin/openood/blob/HEAD/openood/losses/reweight.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d8339d591bbd7f52"}},{"code_sha256_prefix":"61492b343324d2ea","entry":"soft_cross_entropy","repo":"antoinedemathelin/openood","repo_kind":"listed","path":"openood/losses/sce.py","file_url":"https://github.com/antoinedemathelin/openood/blob/HEAD/openood/losses/sce.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"61492b343324d2ea"}},{"code_sha256_prefix":"7023eb86801bb3a3","entry":"tf2th","repo":"antoinedemathelin/openood","repo_kind":"listed","path":"openood/networks/bit.py","file_url":"https://github.com/antoinedemathelin/openood/blob/HEAD/openood/networks/bit.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7023eb86801bb3a3"}},{"code_sha256_prefix":"3cce4936256bdc33","entry":"Discriminator32","repo":"antoinedemathelin/openood","repo_kind":"listed","path":"openood/networks/arpl_net.py","file_url":"https://github.com/antoinedemathelin/openood/blob/HEAD/openood/networks/arpl_net.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3cce4936256bdc33"}},{"code_sha256_prefix":"797cd0913159403b","entry":"Generator","repo":"antoinedemathelin/openood","repo_kind":"listed","path":"openood/networks/arpl_net.py","file_url":"https://github.com/antoinedemathelin/openood/blob/HEAD/openood/networks/arpl_net.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"797cd0913159403b"}},{"code_sha256_prefix":"893f424fd19a9b6c","entry":"Generator32","repo":"antoinedemathelin/openood","repo_kind":"listed","path":"openood/networks/arpl_net.py","file_url":"https://github.com/antoinedemathelin/openood/blob/HEAD/openood/networks/arpl_net.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"893f424fd19a9b6c"}},{"code_sha256_prefix":"45b6edb2cf78cda2","entry":"create_window","repo":"antoinedemathelin/openood","repo_kind":"listed","path":"openood/losses/ssim.py","file_url":"https://github.com/antoinedemathelin/openood/blob/HEAD/openood/losses/ssim.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"45b6edb2cf78cda2"}},{"code_sha256_prefix":"e2424ce033ba3a0f","entry":"gaussian","repo":"antoinedemathelin/openood","repo_kind":"listed","path":"openood/losses/ssim.py","file_url":"https://github.com/antoinedemathelin/openood/blob/HEAD/openood/losses/ssim.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e2424ce033ba3a0f"}},{"code_sha256_prefix":"3fec666ee416a824","entry":"get_shift_module","repo":"antoinedemathelin/openood","repo_kind":"listed","path":"openood/networks/csi_net.py","file_url":"https://github.com/antoinedemathelin/openood/blob/HEAD/openood/networks/csi_net.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3fec666ee416a824"}},{"code_sha256_prefix":"4c1d0e3674921db5","entry":"loss_function","repo":"antoinedemathelin/openood","repo_kind":"listed","path":"openood/losses/rd4ad_loss.py","file_url":"https://github.com/antoinedemathelin/openood/blob/HEAD/openood/losses/rd4ad_loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4c1d0e3674921db5"}},{"code_sha256_prefix":"e1014da2309b3946","entry":"rew_sce","repo":"antoinedemathelin/openood","repo_kind":"listed","path":"openood/losses/reweight.py","file_url":"https://github.com/antoinedemathelin/openood/blob/HEAD/openood/losses/reweight.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e1014da2309b3946"}},{"code_sha256_prefix":"d1619ad1bb7ea0f7","entry":"ssim","repo":"antoinedemathelin/openood","repo_kind":"listed","path":"openood/losses/ssim.py","file_url":"https://github.com/antoinedemathelin/openood/blob/HEAD/openood/losses/ssim.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d1619ad1bb7ea0f7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}