{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/types-of-out-of-distribution-texts-and-how-to","title":"Types of Out-of-Distribution Texts and How to Detect Them","arxiv_id":"2109.06827","date":"2021-09-14","proceeding":"EMNLP 2021 11","authors":["Udit Arora","William Huang","He He"],"abstract":"Despite agreement on the importance of detecting out-of-distribution (OOD) examples, there is little consensus on the formal definition of OOD examples and how to best detect them. We categorize these examples by whether they exhibit a background shift or a semantic shift, and find that the two major approaches to OOD detection, model calibration and density estimation (language modeling for text), have distinct behavior on these types of OOD data. Across 14 pairs of in-distribution and OOD English natural language understanding datasets, we find that density estimation methods consistently beat calibration methods in background shift settings, while performing worse in semantic shift settings. In addition, we find that both methods generally fail to detect examples from challenge data, highlighting a weak spot for current methods. Since no single method works well across all settings, our results call for an explicit definition of OOD examples when evaluating different detection methods.","url_abs":"https://arxiv.org/abs/2109.06827v2","url_pdf":"https://arxiv.org/pdf/2109.06827v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"types-of-out-of-distribution-texts-and-how-to","repo_url":"https://github.com/uditarora/ood-text-emnlp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"density-estimation","task_name":"Density Estimation"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"natural-language-understanding","task_name":"Natural Language Understanding"},{"task_slug":"ood-detection","task_name":"Out of Distribution (OOD) Detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2109.06827","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2109.06827"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/uditarora/ood-text-emnlp","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":12},"by_repo_kind":{"official":{"samples":12,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f3167e070ba7c977","entry":"compute_all","repo":"uditarora/ood-text-emnlp","repo_kind":"official","path":"perplexity.py","file_url":"https://github.com/uditarora/ood-text-emnlp/blob/HEAD/perplexity.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f3167e070ba7c977"}},{"code_sha256_prefix":"72563a8679267472","entry":"compute_auroc","repo":"uditarora/ood-text-emnlp","repo_kind":"official","path":"utils.py","file_url":"https://github.com/uditarora/ood-text-emnlp/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"72563a8679267472"}},{"code_sha256_prefix":"4bd8eb730cf38934","entry":"compute_far","repo":"uditarora/ood-text-emnlp","repo_kind":"official","path":"utils.py","file_url":"https://github.com/uditarora/ood-text-emnlp/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4bd8eb730cf38934"}},{"code_sha256_prefix":"b29cddb5f632a9d9","entry":"compute_perplexity","repo":"uditarora/ood-text-emnlp","repo_kind":"official","path":"perplexity.py","file_url":"https://github.com/uditarora/ood-text-emnlp/blob/HEAD/perplexity.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b29cddb5f632a9d9"}},{"code_sha256_prefix":"83b21e96c79e998a","entry":"compute_px","repo":"uditarora/ood-text-emnlp","repo_kind":"official","path":"utils.py","file_url":"https://github.com/uditarora/ood-text-emnlp/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"83b21e96c79e998a"}},{"code_sha256_prefix":"f9a4adc707cb9193","entry":"encode","repo":"uditarora/ood-text-emnlp","repo_kind":"official","path":"msp_eval.py","file_url":"https://github.com/uditarora/ood-text-emnlp/blob/HEAD/msp_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9a4adc707cb9193"}},{"code_sha256_prefix":"b41e11872936971a","entry":"get_dataloader","repo":"uditarora/ood-text-emnlp","repo_kind":"official","path":"roberta_fine_tune.py","file_url":"https://github.com/uditarora/ood-text-emnlp/blob/HEAD/roberta_fine_tune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b41e11872936971a"}},{"code_sha256_prefix":"30912a8b3a57ba75","entry":"process_custom_dataset","repo":"uditarora/ood-text-emnlp","repo_kind":"official","path":"roberta_fine_tune.py","file_url":"https://github.com/uditarora/ood-text-emnlp/blob/HEAD/roberta_fine_tune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30912a8b3a57ba75"}},{"code_sha256_prefix":"374d3df6220579b7","entry":"process_entailment","repo":"uditarora/ood-text-emnlp","repo_kind":"official","path":"msp_eval.py","file_url":"https://github.com/uditarora/ood-text-emnlp/blob/HEAD/msp_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"374d3df6220579b7"}},{"code_sha256_prefix":"d33fb2634b456ea3","entry":"process_hf_dataset","repo":"uditarora/ood-text-emnlp","repo_kind":"official","path":"roberta_fine_tune.py","file_url":"https://github.com/uditarora/ood-text-emnlp/blob/HEAD/roberta_fine_tune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d33fb2634b456ea3"}},{"code_sha256_prefix":"87d7cfb062dec677","entry":"process_msp","repo":"uditarora/ood-text-emnlp","repo_kind":"official","path":"msp_eval.py","file_url":"https://github.com/uditarora/ood-text-emnlp/blob/HEAD/msp_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"87d7cfb062dec677"}},{"code_sha256_prefix":"0975a4d4fcffe722","entry":"setup","repo":"uditarora/ood-text-emnlp","repo_kind":"official","path":"perplexity.py","file_url":"https://github.com/uditarora/ood-text-emnlp/blob/HEAD/perplexity.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0975a4d4fcffe722"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}