{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-noisy-elephant-in-the-room-is-your-out-of","title":"A noisy elephant in the room: Is your out-of-distribution detector robust to label noise?","arxiv_id":"2404.01775","date":"2024-04-02","proceeding":"CVPR 2024 1","authors":["Galadrielle Humblot-Renaux","Sergio Escalera","Thomas B. Moeslund"],"abstract":"The ability to detect unfamiliar or unexpected images is essential for safe deployment of computer vision systems. In the context of classification, the task of detecting images outside of a model's training domain is known as out-of-distribution (OOD) detection. While there has been a growing research interest in developing post-hoc OOD detection methods, there has been comparably little discussion around how these methods perform when the underlying classifier is not trained on a clean, carefully curated dataset. In this work, we take a closer look at 20 state-of-the-art OOD detection methods in the (more realistic) scenario where the labels used to train the underlying classifier are unreliable (e.g. crowd-sourced or web-scraped labels). Extensive experiments across different datasets, noise types & levels, architectures and checkpointing strategies provide insights into the effect of class label noise on OOD detection, and show that poor separation between incorrectly classified ID samples vs. OOD samples is an overlooked yet important limitation of existing methods. Code: https://github.com/glhr/ood-labelnoise","url_abs":"https://arxiv.org/abs/2404.01775v1","url_pdf":"https://arxiv.org/pdf/2404.01775v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-noisy-elephant-in-the-room-is-your-out-of","repo_url":"https://github.com/glhr/ood-labelnoise","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"GPL-3.0"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification-with-label-noise","task_name":"Image Classification with Label Noise"},{"task_slug":"ood-detection","task_name":"Out of Distribution (OOD) Detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2404.01775","atlas_url":"https://app.syntology.ai/?focus=2404.01775","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.01775"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/glhr/ood-labelnoise","reach":{"status":"ok","spdx":"GPL-3.0"}}],"summary":{"ran":7,"ran_violates":1,"unverified":3},"by_repo_kind":{"official":{"samples":11,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"0b07b2fc0c1761bf","entry":"calc_standard_metrics","repo":"glhr/ood-labelnoise","repo_kind":"official","path":"analysis/ood_metrics.py","file_url":"https://github.com/glhr/ood-labelnoise/blob/HEAD/analysis/ood_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"0b07b2fc0c1761bf"}},{"code_sha256_prefix":"9d1ae0e9b04ef642","entry":"fnr_at_tnr","repo":"glhr/ood-labelnoise","repo_kind":"official","path":"analysis/ood_metrics.py","file_url":"https://github.com/glhr/ood-labelnoise/blob/HEAD/analysis/ood_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"9d1ae0e9b04ef642"}},{"code_sha256_prefix":"2db3e4a0a7bcaae9","entry":"fpr_at_tpr","repo":"glhr/ood-labelnoise","repo_kind":"official","path":"analysis/ood_metrics.py","file_url":"https://github.com/glhr/ood-labelnoise/blob/HEAD/analysis/ood_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"2db3e4a0a7bcaae9"}},{"code_sha256_prefix":"5623ccf1107cb331","entry":"get_noise_rates","repo":"glhr/ood-labelnoise","repo_kind":"official","path":"analysis/results_utils.py","file_url":"https://github.com/glhr/ood-labelnoise/blob/HEAD/analysis/results_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"5623ccf1107cb331"}},{"code_sha256_prefix":"70177e3540b00fe1","entry":"group_noise","repo":"glhr/ood-labelnoise","repo_kind":"official","path":"analysis/data_beautification.py","file_url":"https://github.com/glhr/ood-labelnoise/blob/HEAD/analysis/data_beautification.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"70177e3540b00fe1"}},{"code_sha256_prefix":"5ea217b0749899cf","entry":"noisy_or_not","repo":"glhr/ood-labelnoise","repo_kind":"official","path":"analysis/data_beautification.py","file_url":"https://github.com/glhr/ood-labelnoise/blob/HEAD/analysis/data_beautification.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"5ea217b0749899cf"}},{"code_sha256_prefix":"ffed19ae9988f74a","entry":"rename_iid_ds","repo":"glhr/ood-labelnoise","repo_kind":"official","path":"analysis/data_beautification.py","file_url":"https://github.com/glhr/ood-labelnoise/blob/HEAD/analysis/data_beautification.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"ffed19ae9988f74a"}},{"code_sha256_prefix":"d8339d591bbd7f52","entry":"rew_ce","repo":"glhr/ood-labelnoise","repo_kind":"official","path":"openood/losses/reweight.py","file_url":"https://github.com/glhr/ood-labelnoise/blob/HEAD/openood/losses/reweight.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"d8339d591bbd7f52"}},{"code_sha256_prefix":"9f961af660314c45","entry":"get_res_from_scores","repo":"glhr/ood-labelnoise","repo_kind":"official","path":"analysis/results_utils.py","file_url":"https://github.com/glhr/ood-labelnoise/blob/HEAD/analysis/results_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"9f961af660314c45"}},{"code_sha256_prefix":"4c1d0e3674921db5","entry":"loss_function","repo":"glhr/ood-labelnoise","repo_kind":"official","path":"openood/losses/rd4ad_loss.py","file_url":"https://github.com/glhr/ood-labelnoise/blob/HEAD/openood/losses/rd4ad_loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"4c1d0e3674921db5"}},{"code_sha256_prefix":"e1014da2309b3946","entry":"rew_sce","repo":"glhr/ood-labelnoise","repo_kind":"official","path":"openood/losses/reweight.py","file_url":"https://github.com/glhr/ood-labelnoise/blob/HEAD/openood/losses/reweight.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"e1014da2309b3946"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}