{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/contrastive-out-of-distribution-detection-for","title":"Contrastive Out-of-Distribution Detection for Pretrained Transformers","arxiv_id":"2104.08812","date":"2021-04-18","proceeding":"EMNLP 2021 11","authors":["Wenxuan Zhou","Fangyu Liu","Muhao Chen"],"abstract":"Pretrained Transformers achieve remarkable performance when training and test data are from the same distribution. However, in real-world scenarios, the model often faces out-of-distribution (OOD) instances that can cause severe semantic shift problems at inference time. Therefore, in practice, a reliable model should identify such instances, and then either reject them during inference or pass them over to models that handle another distribution. In this paper, we develop an unsupervised OOD detection method, in which only the in-distribution (ID) data are used in training. We propose to fine-tune the Transformers with a contrastive loss, which improves the compactness of representations, such that OOD instances can be better differentiated from ID ones. These OOD instances can then be accurately detected using the Mahalanobis distance in the model's penultimate layer. We experiment with comprehensive settings and achieve near-perfect OOD detection performance, outperforming baselines drastically. We further investigate the rationales behind the improvement, finding that more compact representations through margin-based contrastive learning bring the improvement. We release our code to the community for future research.","url_abs":"https://arxiv.org/abs/2104.08812v2","url_pdf":"https://arxiv.org/pdf/2104.08812v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"contrastive-out-of-distribution-detection-for","repo_url":"https://github.com/wzhouad/Contra-OOD","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"ood-detection","task_name":"Out of Distribution (OOD) Detection"},{"task_slug":"out-of-distribution-detection","task_name":"Out-of-Distribution Detection"}],"methods":[{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2104.08812","atlas_url":"https://app.syntology.ai/?focus=2104.08812","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2104.08812"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wzhouad/Contra-OOD","reach":null}],"summary":{"ran_draft_wrong":2,"ran_fixture":3,"ran_honours":1,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"fba9a666b9277255","entry":"collate_fn","repo":"wzhouad/Contra-OOD","repo_kind":"official","path":"evaluation.py","file_url":"https://github.com/wzhouad/Contra-OOD/blob/HEAD/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fba9a666b9277255"}},{"code_sha256_prefix":"541b471fca0ca15a","entry":"fpr_and_fdr_at_recall","repo":"wzhouad/Contra-OOD","repo_kind":"official","path":"evaluation.py","file_url":"https://github.com/wzhouad/Contra-OOD/blob/HEAD/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"541b471fca0ca15a"}},{"code_sha256_prefix":"ace38fb975eb5cfa","entry":"get_auroc","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"ace38fb975eb5cfa"}},{"code_sha256_prefix":"b20471eaf133a65e","entry":"get_fpr_95","repo":"wzhouad/Contra-OOD","repo_kind":"official","path":"evaluation.py","file_url":"https://github.com/wzhouad/Contra-OOD/blob/HEAD/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b20471eaf133a65e"}},{"code_sha256_prefix":"21a8e085a4502dad","entry":"merge_keys","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"21a8e085a4502dad"}},{"code_sha256_prefix":"e6809651fe876f1d","entry":"stable_cumsum","repo":"wzhouad/Contra-OOD","repo_kind":"official","path":"evaluation.py","file_url":"https://github.com/wzhouad/Contra-OOD/blob/HEAD/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e6809651fe876f1d"}},{"code_sha256_prefix":"b1476bb643081c90","entry":"evaluate_ood","repo":"wzhouad/Contra-OOD","repo_kind":"official","path":"evaluation.py","file_url":"https://github.com/wzhouad/Contra-OOD/blob/HEAD/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1476bb643081c90"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}