{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ad-llm-benchmarking-large-language-models-for","title":"AD-LLM: Benchmarking Large Language Models for Anomaly Detection","arxiv_id":"2412.11142","date":"2024-12-15","proceeding":null,"authors":["Tiankai Yang","Yi Nian","Shawn Li","Ruiyao Xu","Yuangang Li","Jiaqi Li","Zhuo Xiao","Xiyang Hu","Ryan Rossi","Kaize Ding","Xia Hu","Yue Zhao"],"abstract":"Anomaly detection (AD) is an important machine learning task with many real-world uses, including fraud detection, medical diagnosis, and industrial monitoring. Within natural language processing (NLP), AD helps detect issues like spam, misinformation, and unusual user activity. Although large language models (LLMs) have had a strong impact on tasks such as text generation and summarization, their potential in AD has not been studied enough. This paper introduces AD-LLM, the first benchmark that evaluates how LLMs can help with NLP anomaly detection. We examine three key tasks: (i) zero-shot detection, using LLMs' pre-trained knowledge to perform AD without tasks-specific training; (ii) data augmentation, generating synthetic data and category descriptions to improve AD models; and (iii) model selection, using LLMs to suggest unsupervised AD models. Through experiments with different datasets, we find that LLMs can work well in zero-shot AD, that carefully designed augmentation methods are useful, and that explaining model selection for specific datasets remains challenging. Based on these results, we outline six future research directions on LLMs for AD.","url_abs":"https://arxiv.org/abs/2412.11142v3","url_pdf":"https://arxiv.org/pdf/2412.11142v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ad-llm-benchmarking-large-language-models-for","repo_url":"https://github.com/usc-fortis/ad-llm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"ad-llm-benchmarking-large-language-models-for","repo_url":"https://github.com/yzhao062/pyod","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"anomaly-detection","task_name":"Anomaly Detection"},{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"fraud-detection","task_name":"Fraud Detection"},{"task_slug":"medical-diagnosis","task_name":"Medical Diagnosis"},{"task_slug":"misinformation","task_name":"Misinformation"},{"task_slug":"model-selection","task_name":"Model Selection"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.11142","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.11142"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yzhao062/pyod","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/usc-fortis/ad-llm","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":7},"by_repo_kind":{"official":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b45367566abc496b","entry":"generate_embeddings","repo":"usc-fortis/ad-llm","repo_kind":"official","path":"baseline_w_gpt_embed.py","file_url":"https://github.com/usc-fortis/ad-llm/blob/HEAD/baseline_w_gpt_embed.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b45367566abc496b"}},{"code_sha256_prefix":"141469aa23018895","entry":"generate_prompt_setting_1","repo":"usc-fortis/ad-llm","repo_kind":"official","path":"prompt/ad_1_gpt_prompt.py","file_url":"https://github.com/usc-fortis/ad-llm/blob/HEAD/prompt/ad_1_gpt_prompt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"141469aa23018895"}},{"code_sha256_prefix":"8aa0ff2182ce2c41","entry":"generate_prompt_setting_1","repo":"usc-fortis/ad-llm","repo_kind":"official","path":"prompt/ad_1_llama_prompt.py","file_url":"https://github.com/usc-fortis/ad-llm/blob/HEAD/prompt/ad_1_llama_prompt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8aa0ff2182ce2c41"}},{"code_sha256_prefix":"d781fa3b578a72e3","entry":"generate_prompt_setting_2","repo":"usc-fortis/ad-llm","repo_kind":"official","path":"prompt/ad_2_gpt_prompt.py","file_url":"https://github.com/usc-fortis/ad-llm/blob/HEAD/prompt/ad_2_gpt_prompt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d781fa3b578a72e3"}},{"code_sha256_prefix":"8350c5663464101b","entry":"read_data_summary","repo":"usc-fortis/ad-llm","repo_kind":"official","path":"utils.py","file_url":"https://github.com/usc-fortis/ad-llm/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8350c5663464101b"}},{"code_sha256_prefix":"772778f25f3bd45f","entry":"read_json","repo":"usc-fortis/ad-llm","repo_kind":"official","path":"utils.py","file_url":"https://github.com/usc-fortis/ad-llm/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"772778f25f3bd45f"}},{"code_sha256_prefix":"abc3be5d5ec0f373","entry":"read_normal_desc","repo":"usc-fortis/ad-llm","repo_kind":"official","path":"utils.py","file_url":"https://github.com/usc-fortis/ad-llm/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"abc3be5d5ec0f373"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}