{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/active-anomaly-detection-via-ensembles-1","title":"Effectiveness of Tree-based Ensembles for Anomaly Discovery: Insights, Batch and Streaming Active Learning","arxiv_id":"1901.08930","date":"2019-01-23","proceeding":null,"authors":["Shubhomoy Das","Md. Rakibul Islam","Nitthilan Kannappan Jayakodi","Janardhan Rao Doppa"],"abstract":"In many real-world AD applications including computer security and fraud prevention, the anomaly detector must be configurable by the human analyst to minimize the effort on false positives. One important way to configure the detector is by providing true labels (nominal or anomaly) for a few instances. Recent work on active anomaly discovery has shown that greedily querying the top-scoring instance and tuning the weights of ensemble detectors based on label feedback allows us to quickly discover true anomalies. This paper makes four main contributions to improve the state-of-the-art in anomaly discovery using tree-based ensembles. First, we provide an important insight that explains the practical successes of unsupervised tree-based ensembles and active learning based on greedy query selection strategy. We also present empirical results on real-world data to support our insights and theoretical analysis to support active learning. Second, we develop a novel batch active learning algorithm to improve the diversity of discovered anomalies based on a formalism called compact description to describe the discovered anomalies. Third, we develop a novel active learning algorithm to handle streaming data setting. We present a data drift detection algorithm that not only detects the drift robustly, but also allows us to take corrective actions to adapt the anomaly detector in a principled manner. Fourth, we present extensive experiments to evaluate our insights and our tree-based active anomaly discovery algorithms in both batch and streaming data settings. Our results show that active learning allows us to discover significantly more anomalies than state-of-the-art unsupervised baselines, our batch active learning algorithm discovers diverse anomalies, and our algorithms under the streaming-data setup are competitive with the batch setup.","url_abs":"https://arxiv.org/abs/1901.08930v3","url_pdf":"https://arxiv.org/pdf/1901.08930v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"active-anomaly-detection-via-ensembles-1","repo_url":"https://github.com/shubhomoydas/ad_examples","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"active-anomaly-detection-via-ensembles-1","repo_url":"https://github.com/freedombenLiu/ad_examples","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"active-learning","task_name":"Active Learning"},{"task_slug":"anomaly-detection","task_name":"Anomaly Detection"},{"task_slug":"computer-security","task_name":"Computer Security"},{"task_slug":"drift-detection","task_name":"Drift Detection"}],"methods":[{"method_slug":"interpretability","method_name":"Interpretability"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1901.08930","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1901.08930"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shubhomoydas/ad_examples","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/freedombenLiu/ad_examples","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":21},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1},"listed":{"samples":16,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8badd090540c89ba","entry":"aad_loss_gradient_linear","repo":"freedombenLiu/ad_examples","repo_kind":"listed","path":"ad_examples/aad/aad_loss.py","file_url":"https://github.com/freedombenLiu/ad_examples/blob/HEAD/ad_examples/aad/aad_loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8badd090540c89ba"}},{"code_sha256_prefix":"c14e158c80a69604","entry":"aad_loss_linear","repo":"freedombenLiu/ad_examples","repo_kind":"listed","path":"ad_examples/aad/aad_loss.py","file_url":"https://github.com/freedombenLiu/ad_examples/blob/HEAD/ad_examples/aad/aad_loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c14e158c80a69604"}},{"code_sha256_prefix":"151fc694a94564e2","entry":"describe_instances","repo":"shubhomoydas/ad_examples","repo_kind":"official","path":"ad_examples/aad/demo_aad.py","file_url":"https://github.com/shubhomoydas/ad_examples/blob/HEAD/ad_examples/aad/demo_aad.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"151fc694a94564e2"}},{"code_sha256_prefix":"f958c10e9cffa78e","entry":"detect_anomalies_and_describe","repo":"shubhomoydas/ad_examples","repo_kind":"official","path":"ad_examples/aad/demo_aad.py","file_url":"https://github.com/shubhomoydas/ad_examples/blob/HEAD/ad_examples/aad/demo_aad.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f958c10e9cffa78e"}},{"code_sha256_prefix":"eb82e43983ab9524","entry":"estimate_qtau","repo":"freedombenLiu/ad_examples","repo_kind":"listed","path":"ad_examples/aad/aad_base.py","file_url":"https://github.com/freedombenLiu/ad_examples/blob/HEAD/ad_examples/aad/aad_base.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eb82e43983ab9524"}},{"code_sha256_prefix":"f83d93a9883765bf","entry":"evaluate_forest_original","repo":"freedombenLiu/ad_examples","repo_kind":"listed","path":"ad_examples/aad/aad_test_support.py","file_url":"https://github.com/freedombenLiu/ad_examples/blob/HEAD/ad_examples/aad/aad_test_support.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f83d93a9883765bf"}},{"code_sha256_prefix":"e3cbbcf9de858a91","entry":"get_aad_command_args","repo":"freedombenLiu/ad_examples","repo_kind":"listed","path":"ad_examples/aad/aad_globals.py","file_url":"https://github.com/freedombenLiu/ad_examples/blob/HEAD/ad_examples/aad/aad_globals.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e3cbbcf9de858a91"}},{"code_sha256_prefix":"f3f7f564b0d36cdb","entry":"get_aad_metrics_structure","repo":"freedombenLiu/ad_examples","repo_kind":"listed","path":"ad_examples/aad/aad_base.py","file_url":"https://github.com/freedombenLiu/ad_examples/blob/HEAD/ad_examples/aad/aad_base.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f3f7f564b0d36cdb"}},{"code_sha256_prefix":"a7f5521ccb2189dd","entry":"get_aad_model","repo":"freedombenLiu/ad_examples","repo_kind":"listed","path":"ad_examples/aad/aad_support.py","file_url":"https://github.com/freedombenLiu/ad_examples/blob/HEAD/ad_examples/aad/aad_support.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a7f5521ccb2189dd"}},{"code_sha256_prefix":"be92dddc5f891292","entry":"get_budget_topK","repo":"freedombenLiu/ad_examples","repo_kind":"listed","path":"ad_examples/aad/aad_base.py","file_url":"https://github.com/freedombenLiu/ad_examples/blob/HEAD/ad_examples/aad/aad_base.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"be92dddc5f891292"}},{"code_sha256_prefix":"8693d09b634b4c69","entry":"get_first_val_not_marked","repo":"freedombenLiu/ad_examples","repo_kind":"listed","path":"ad_examples/aad/aad_globals.py","file_url":"https://github.com/freedombenLiu/ad_examples/blob/HEAD/ad_examples/aad/aad_globals.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8693d09b634b4c69"}},{"code_sha256_prefix":"407db3dd36ec6a4d","entry":"get_first_vals_not_marked","repo":"freedombenLiu/ad_examples","repo_kind":"listed","path":"ad_examples/aad/aad_globals.py","file_url":"https://github.com/freedombenLiu/ad_examples/blob/HEAD/ad_examples/aad/aad_globals.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"407db3dd36ec6a4d"}},{"code_sha256_prefix":"4f9fdcf28e80b3bc","entry":"get_mean_euclidean_distance","repo":"freedombenLiu/ad_examples","repo_kind":"listed","path":"ad_examples/aad/query_model_euclidean.py","file_url":"https://github.com/freedombenLiu/ad_examples/blob/HEAD/ad_examples/aad/query_model_euclidean.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4f9fdcf28e80b3bc"}},{"code_sha256_prefix":"3aaf622d2fa22ace","entry":"get_min_euclidean_distance","repo":"freedombenLiu/ad_examples","repo_kind":"listed","path":"ad_examples/aad/query_model_euclidean.py","file_url":"https://github.com/freedombenLiu/ad_examples/blob/HEAD/ad_examples/aad/query_model_euclidean.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3aaf622d2fa22ace"}},{"code_sha256_prefix":"2633e06aed91c4d2","entry":"get_rulesets","repo":"freedombenLiu/ad_examples","repo_kind":"listed","path":"ad_examples/aad/aad_ruleset_support.py","file_url":"https://github.com/freedombenLiu/ad_examples/blob/HEAD/ad_examples/aad/aad_ruleset_support.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2633e06aed91c4d2"}},{"code_sha256_prefix":"35491236d4cbf696","entry":"get_score_ranges","repo":"freedombenLiu/ad_examples","repo_kind":"listed","path":"ad_examples/aad/aad_support.py","file_url":"https://github.com/freedombenLiu/ad_examples/blob/HEAD/ad_examples/aad/aad_support.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"35491236d4cbf696"}},{"code_sha256_prefix":"99863fb4df7a185c","entry":"prepare_aad_model","repo":"shubhomoydas/ad_examples","repo_kind":"official","path":"ad_examples/aad/aad_stream.py","file_url":"https://github.com/shubhomoydas/ad_examples/blob/HEAD/ad_examples/aad/aad_stream.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"99863fb4df7a185c"}},{"code_sha256_prefix":"b6e9c8b277dfd2e6","entry":"prepare_conjunctive_rulesets","repo":"freedombenLiu/ad_examples","repo_kind":"listed","path":"ad_examples/aad/aad_ruleset_support.py","file_url":"https://github.com/freedombenLiu/ad_examples/blob/HEAD/ad_examples/aad/aad_ruleset_support.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b6e9c8b277dfd2e6"}},{"code_sha256_prefix":"3cf89ba27b591b36","entry":"prepare_stream_anomaly_detector","repo":"shubhomoydas/ad_examples","repo_kind":"official","path":"ad_examples/aad/aad_stream.py","file_url":"https://github.com/shubhomoydas/ad_examples/blob/HEAD/ad_examples/aad/aad_stream.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3cf89ba27b591b36"}},{"code_sha256_prefix":"61602ce877390332","entry":"summarize_aad_metrics","repo":"freedombenLiu/ad_examples","repo_kind":"listed","path":"ad_examples/aad/aad_support.py","file_url":"https://github.com/freedombenLiu/ad_examples/blob/HEAD/ad_examples/aad/aad_support.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"61602ce877390332"}},{"code_sha256_prefix":"8f255acab32975b3","entry":"train_aad_model","repo":"shubhomoydas/ad_examples","repo_kind":"official","path":"ad_examples/aad/aad_stream.py","file_url":"https://github.com/shubhomoydas/ad_examples/blob/HEAD/ad_examples/aad/aad_stream.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8f255acab32975b3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}