{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sok-pragmatic-assessment-of-machine-learning","title":"SoK: Pragmatic Assessment of Machine Learning for Network Intrusion Detection","arxiv_id":"2305.00550","date":"2023-04-30","proceeding":null,"authors":["Giovanni Apruzzese","Pavel Laskov","Johannes Schneider"],"abstract":"Machine Learning (ML) has become a valuable asset to solve many real-world tasks. For Network Intrusion Detection (NID), however, scientific advances in ML are still seen with skepticism by practitioners. This disconnection is due to the intrinsically limited scope of research papers, many of which primarily aim to demonstrate new methods ``outperforming'' prior work -- oftentimes overlooking the practical implications for deploying the proposed solutions in real systems. Unfortunately, the value of ML for NID depends on a plethora of factors, such as hardware, that are often neglected in scientific literature. This paper aims to reduce the practitioners' skepticism towards ML for NID by \"changing\" the evaluation methodology adopted in research. After elucidating which \"factors\" influence the operational deployment of ML in NID, we propose the notion of \"pragmatic assessment\", which enable practitioners to gauge the real value of ML methods for NID. Then, we show that the state-of-research hardly allows one to estimate the value of ML for NID. As a constructive step forward, we carry out a pragmatic assessment. We re-assess existing ML methods for NID, focusing on the classification of malicious network traffic, and consider: hundreds of configuration settings; diverse adversarial scenarios; and four hardware platforms. Our large and reproducible evaluations enable estimating the quality of ML for NID. We also validate our claims through a user-study with security practitioners.","url_abs":"https://arxiv.org/abs/2305.00550v1","url_pdf":"https://arxiv.org/pdf/2305.00550v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sok-pragmatic-assessment-of-machine-learning","repo_url":"https://github.com/hihey54/pragmaticassessment","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"intrusion-detection","task_name":"Intrusion Detection"},{"task_slug":"network-intrusion-detection","task_name":"Network Intrusion Detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2305.00550","atlas_url":"https://app.syntology.ai/?focus=2305.00550","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.00550"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hihey54/pragmaticassessment","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3b0a6fccb55d8564","entry":"compute_results","repo":"hihey54/pragmaticassessment","repo_kind":"official","path":"pragmatic_assessment/evaluation/supportFunctions.py","file_url":"https://github.com/hihey54/pragmaticassessment/blob/HEAD/pragmatic_assessment/evaluation/supportFunctions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3b0a6fccb55d8564"}},{"code_sha256_prefix":"0dfa34ebc868fc0c","entry":"develop_clf","repo":"hihey54/pragmaticassessment","repo_kind":"official","path":"pragmatic_assessment/evaluation/supportFunctions.py","file_url":"https://github.com/hihey54/pragmaticassessment/blob/HEAD/pragmatic_assessment/evaluation/supportFunctions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0dfa34ebc868fc0c"}},{"code_sha256_prefix":"718e9fb70accb96b","entry":"evaluate_clf","repo":"hihey54/pragmaticassessment","repo_kind":"official","path":"pragmatic_assessment/evaluation/supportFunctions.py","file_url":"https://github.com/hihey54/pragmaticassessment/blob/HEAD/pragmatic_assessment/evaluation/supportFunctions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"718e9fb70accb96b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}