{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/anoshift-a-distribution-shift-benchmark-for","title":"AnoShift: A Distribution Shift Benchmark for Unsupervised Anomaly Detection","arxiv_id":"2206.15476","date":"2022-06-30","proceeding":null,"authors":["Marius Dragoi","Elena Burceanu","Emanuela Haller","Andrei Manolache","Florin Brad"],"abstract":"Analyzing the distribution shift of data is a growing research direction in nowadays Machine Learning (ML), leading to emerging new benchmarks that focus on providing a suitable scenario for studying the generalization properties of ML models. The existing benchmarks are focused on supervised learning, and to the best of our knowledge, there is none for unsupervised learning. Therefore, we introduce an unsupervised anomaly detection benchmark with data that shifts over time, built over Kyoto-2006+, a traffic dataset for network intrusion detection. This type of data meets the premise of shifting the input distribution: it covers a large time span ($10$ years), with naturally occurring changes over time (eg users modifying their behavior patterns, and software updates). We first highlight the non-stationary nature of the data, using a basic per-feature analysis, t-SNE, and an Optimal Transport approach for measuring the overall distribution distances between years. Next, we propose AnoShift, a protocol splitting the data in IID, NEAR, and FAR testing splits. We validate the performance degradation over time with diverse models, ranging from classical approaches to deep learning. Finally, we show that by acknowledging the distribution shift problem and properly addressing it, the performance can be improved compared to the classical training which assumes independent and identically distributed data (on average, by up to $3\\%$ for our approach). Dataset and code are available at https://github.com/bit-ml/AnoShift/.","url_abs":"https://arxiv.org/abs/2206.15476v4","url_pdf":"https://arxiv.org/pdf/2206.15476v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"anoshift-a-distribution-shift-benchmark-for","repo_url":"https://github.com/bit-ml/anoshift","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"intrusion-detection","task_name":"Intrusion Detection"},{"task_slug":"network-intrusion-detection","task_name":"Network Intrusion Detection"},{"task_slug":"unsupervised-anomaly-detection","task_name":"Unsupervised Anomaly Detection"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[{"slug":"anoshift","name":"AnoShift","full_name":"AnoShift: A Distribution Shift Benchmark for Unsupervised Anomaly Detection"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/unsupervised-anomaly-detection-on-anoshift","task":"Unsupervised Anomaly Detection","dataset":"AnoShift","model":"COPOD","rank_in_archive_order":5,"of":15,"metrics":{"ROC-AUC FAR":"50.42","ROC-AUC IID":"85.62","ROC-AUC NEAR":"54.24","ROC-AUC-ID (In-Distribution setup)":"80.89"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-anomaly-detection-on-anoshift","task":"Unsupervised Anomaly Detection","dataset":"AnoShift","model":"OC-SVM","rank_in_archive_order":6,"of":15,"metrics":{"ROC-AUC FAR":"49.57","ROC-AUC IID":"76.86","ROC-AUC NEAR":"71.43","ROC-AUC-ID (In-Distribution setup)":"68.73"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-anomaly-detection-on-anoshift","task":"Unsupervised Anomaly Detection","dataset":"AnoShift","model":"SO-GAAL","rank_in_archive_order":7,"of":15,"metrics":{"ROC-AUC FAR":"49.35","ROC-AUC IID":"50.48","ROC-AUC NEAR":"54.55","ROC-AUC-ID (In-Distribution setup)":"49.90"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-anomaly-detection-on-anoshift","task":"Unsupervised Anomaly Detection","dataset":"AnoShift","model":"ECOD Li et al. (2022)","rank_in_archive_order":8,"of":15,"metrics":{"ROC-AUC FAR":"49.19","ROC-AUC IID":"84.76","ROC-AUC NEAR":"44.87","ROC-AUC-ID (In-Distribution setup)":"79.41"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-anomaly-detection-on-anoshift","task":"Unsupervised Anomaly Detection","dataset":"AnoShift","model":"LOF","rank_in_archive_order":9,"of":15,"metrics":{"ROC-AUC FAR":"34.96","ROC-AUC IID":"91.5","ROC-AUC NEAR":"79.29","ROC-AUC-ID (In-Distribution setup)":"87.61"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-anomaly-detection-on-anoshift","task":"Unsupervised Anomaly Detection","dataset":"AnoShift","model":"deepSVDD","rank_in_archive_order":10,"of":15,"metrics":{"ROC-AUC FAR":"34.53","ROC-AUC IID":"92.67","ROC-AUC NEAR":"87.00","ROC-AUC-ID (In-Distribution setup)":"88.24"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-anomaly-detection-on-anoshift","task":"Unsupervised Anomaly Detection","dataset":"AnoShift","model":"LUNAR","rank_in_archive_order":11,"of":15,"metrics":{"ROC-AUC FAR":"28.19","ROC-AUC IID":"85.75","ROC-AUC NEAR":"49.03","ROC-AUC-ID (In-Distribution setup)":"78.53"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-anomaly-detection-on-anoshift","task":"Unsupervised Anomaly Detection","dataset":"AnoShift","model":"BERT","rank_in_archive_order":12,"of":15,"metrics":{"ROC-AUC FAR":"28.15","ROC-AUC IID":"84.54","ROC-AUC NEAR":"86.05","ROC-AUC-ID (In-Distribution setup)":"79.62"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-anomaly-detection-on-anoshift","task":"Unsupervised Anomaly Detection","dataset":"AnoShift","model":"IsoForest","rank_in_archive_order":13,"of":15,"metrics":{"ROC-AUC FAR":"27.16","ROC-AUC IID":"86.09","ROC-AUC NEAR":"75.26","ROC-AUC-ID (In-Distribution setup)":"81.27"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-anomaly-detection-on-anoshift","task":"Unsupervised Anomaly Detection","dataset":"AnoShift","model":"Internal Contrastive Learning","rank_in_archive_order":14,"of":15,"metrics":{"ROC-AUC FAR":"22.45","ROC-AUC IID":"84.86","ROC-AUC NEAR":"52.26","ROC-AUC-ID (In-Distribution setup)":"66.99"},"uses_additional_data":false},{"leaderboard":"/sota/unsupervised-anomaly-detection-on-anoshift","task":"Unsupervised Anomaly Detection","dataset":"AnoShift","model":"AE for anomalies","rank_in_archive_order":15,"of":15,"metrics":{"ROC-AUC FAR":"19.96","ROC-AUC IID":"81","ROC-AUC NEAR":"44.06","ROC-AUC-ID (In-Distribution setup)":"64.08"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2206.15476","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.15476"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bit-ml/anoshift","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"unverified":13},"by_repo_kind":{"official":{"samples":13,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ab5ca7b7355911f2","entry":"anomaly_score_bert","repo":"bit-ml/anoshift","repo_kind":"official","path":"language_models/evaluation_utils.py","file_url":"https://github.com/bit-ml/anoshift/blob/HEAD/language_models/evaluation_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"ab5ca7b7355911f2"}},{"code_sha256_prefix":"5a538b08c1e8471b","entry":"configure_model","repo":"bit-ml/anoshift","repo_kind":"official","path":"language_models/model_utils.py","file_url":"https://github.com/bit-ml/anoshift/blob/HEAD/language_models/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"5a538b08c1e8471b"}},{"code_sha256_prefix":"9122185befcd3a48","entry":"configure_tokenizer","repo":"bit-ml/anoshift","repo_kind":"official","path":"language_models/tokenizer_utils.py","file_url":"https://github.com/bit-ml/anoshift/blob/HEAD/language_models/tokenizer_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"9122185befcd3a48"}},{"code_sha256_prefix":"d2e17a1b030b09e0","entry":"eval_rocauc_ds","repo":"bit-ml/anoshift","repo_kind":"official","path":"language_models/evaluation_utils.py","file_url":"https://github.com/bit-ml/anoshift/blob/HEAD/language_models/evaluation_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"d2e17a1b030b09e0"}},{"code_sha256_prefix":"67544ef8ead6799f","entry":"load_test_year","repo":"bit-ml/anoshift","repo_kind":"official","path":"baselines_ID_setup/load_anoshift.py","file_url":"https://github.com/bit-ml/anoshift/blob/HEAD/baselines_ID_setup/load_anoshift.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"67544ef8ead6799f"}},{"code_sha256_prefix":"d7ad456b5c436b34","entry":"load_test_year","repo":"bit-ml/anoshift","repo_kind":"official","path":"baselines_OOD_setup/load_anoshift.py","file_url":"https://github.com/bit-ml/anoshift/blob/HEAD/baselines_OOD_setup/load_anoshift.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"d7ad456b5c436b34"}},{"code_sha256_prefix":"9cefe75d6c8660e7","entry":"load_train_year","repo":"bit-ml/anoshift","repo_kind":"official","path":"baselines_ID_setup/load_anoshift.py","file_url":"https://github.com/bit-ml/anoshift/blob/HEAD/baselines_ID_setup/load_anoshift.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"9cefe75d6c8660e7"}},{"code_sha256_prefix":"5e56d43d0e9f0785","entry":"load_train_year","repo":"bit-ml/anoshift","repo_kind":"official","path":"baselines_OOD_setup/load_anoshift.py","file_url":"https://github.com/bit-ml/anoshift/blob/HEAD/baselines_OOD_setup/load_anoshift.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"5e56d43d0e9f0785"}},{"code_sha256_prefix":"65311fdf0a3bbd77","entry":"prepare_set","repo":"bit-ml/anoshift","repo_kind":"official","path":"baselines_ID_setup/baseline_BERT.py","file_url":"https://github.com/bit-ml/anoshift/blob/HEAD/baselines_ID_setup/baseline_BERT.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"65311fdf0a3bbd77"}},{"code_sha256_prefix":"2d42553a00d0c824","entry":"prepare_test_ds_split","repo":"bit-ml/anoshift","repo_kind":"official","path":"language_models/data_utils.py","file_url":"https://github.com/bit-ml/anoshift/blob/HEAD/language_models/data_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"2d42553a00d0c824"}},{"code_sha256_prefix":"c44561799320572b","entry":"rename_columns","repo":"bit-ml/anoshift","repo_kind":"official","path":"baselines_ID_setup/load_anoshift.py","file_url":"https://github.com/bit-ml/anoshift/blob/HEAD/baselines_ID_setup/load_anoshift.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"c44561799320572b"}},{"code_sha256_prefix":"184f64ac20617dd0","entry":"tokenize_function","repo":"bit-ml/anoshift","repo_kind":"official","path":"language_models/data_utils.py","file_url":"https://github.com/bit-ml/anoshift/blob/HEAD/language_models/data_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"184f64ac20617dd0"}},{"code_sha256_prefix":"750b967b13d190fc","entry":"train_df_to_ds","repo":"bit-ml/anoshift","repo_kind":"official","path":"language_models/data_utils.py","file_url":"https://github.com/bit-ml/anoshift/blob/HEAD/language_models/data_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"750b967b13d190fc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}