{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/classical-out-of-distribution-detection","title":"Classical Out-of-Distribution Detection Methods Benchmark in Text Classification Tasks","arxiv_id":"2307.07002","date":"2023-07-13","proceeding":null,"authors":["Mateusz Baran","Joanna Baran","Mateusz Wójcik","Maciej Zięba","Adam Gonczarek"],"abstract":"State-of-the-art models can perform well in controlled environments, but they often struggle when presented with out-of-distribution (OOD) examples, making OOD detection a critical component of NLP systems. In this paper, we focus on highlighting the limitations of existing approaches to OOD detection in NLP. Specifically, we evaluated eight OOD detection methods that are easily integrable into existing NLP systems and require no additional OOD data or model modifications. One of our contributions is providing a well-structured research environment that allows for full reproducibility of the results. Additionally, our analysis shows that existing OOD detection methods for NLP tasks are not yet sufficiently sensitive to capture all samples characterized by various types of distributional shifts. Particularly challenging testing scenarios arise in cases of background shift and randomly shuffled word order within in domain texts. This highlights the need for future work to develop more effective OOD detection approaches for the NLP problems, and our work provides a well-defined foundation for further research in this area.","url_abs":"https://arxiv.org/abs/2307.07002v1","url_pdf":"https://arxiv.org/pdf/2307.07002v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"classical-out-of-distribution-detection","repo_url":"https://github.com/mateuszbaransanok/trustworthyai","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"out-of-distribution-detection","task_name":"Out-of-Distribution Detection"},{"task_slug":"text-classification","task_name":"Text Classification"},{"task_slug":"text-classification-1","task_name":"text-classification"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2307.07002","atlas_url":"https://app.syntology.ai/?focus=2307.07002","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.07002"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mateuszbaransanok/trustworthyai","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/mateuszbaransanok/TrustworthyAI","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"93204434ae2d97ca","entry":"fpr","repo":"mateuszbaransanok/TrustworthyAI","repo_kind":"official","path":"trustworthyai/utils/metrics.py","file_url":"https://github.com/mateuszbaransanok/TrustworthyAI/blob/HEAD/trustworthyai/utils/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"93204434ae2d97ca"}},{"code_sha256_prefix":"326df5323a1a9812","entry":"get_checkpoint_epoch","repo":"mateuszbaransanok/TrustworthyAI","repo_kind":"official","path":"trustworthyai/utils/checkpoints.py","file_url":"https://github.com/mateuszbaransanok/TrustworthyAI/blob/HEAD/trustworthyai/utils/checkpoints.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"326df5323a1a9812"}},{"code_sha256_prefix":"8e63b8dff24f870b","entry":"get_checkpoint_path","repo":"mateuszbaransanok/TrustworthyAI","repo_kind":"official","path":"trustworthyai/utils/checkpoints.py","file_url":"https://github.com/mateuszbaransanok/TrustworthyAI/blob/HEAD/trustworthyai/utils/checkpoints.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8e63b8dff24f870b"}},{"code_sha256_prefix":"6f8a5d1694970c5a","entry":"auc","repo":"mateuszbaransanok/TrustworthyAI","repo_kind":"official","path":"trustworthyai/utils/metrics.py","file_url":"https://github.com/mateuszbaransanok/TrustworthyAI/blob/HEAD/trustworthyai/utils/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6f8a5d1694970c5a"}},{"code_sha256_prefix":"a356e503fd42cd9a","entry":"calc_metrics","repo":"mateuszbaransanok/TrustworthyAI","repo_kind":"official","path":"trustworthyai/utils/metrics.py","file_url":"https://github.com/mateuszbaransanok/TrustworthyAI/blob/HEAD/trustworthyai/utils/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a356e503fd42cd9a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}