{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vulnerability-detection-with-code-language","title":"Vulnerability Detection with Code Language Models: How Far Are We?","arxiv_id":"2403.18624","date":"2024-03-27","proceeding":null,"authors":["Yangruibo Ding","Yanjun Fu","Omniyyah Ibrahim","Chawin Sitawarin","Xinyun Chen","Basel Alomair","David Wagner","Baishakhi Ray","Yizheng Chen"],"abstract":"In the context of the rising interest in code language models (code LMs) and vulnerability detection, we study the effectiveness of code LMs for detecting vulnerabilities. Our analysis reveals significant shortcomings in existing vulnerability datasets, including poor data quality, low label accuracy, and high duplication rates, leading to unreliable model performance in realistic vulnerability detection scenarios. Additionally, the evaluation methods used with these datasets are not representative of real-world vulnerability detection. To address these challenges, we introduce PrimeVul, a new dataset for training and evaluating code LMs for vulnerability detection. PrimeVul incorporates a novel set of data labeling techniques that achieve comparable label accuracy to human-verified benchmarks while significantly expanding the dataset. It also implements a rigorous data de-duplication and chronological data splitting strategy to mitigate data leakage issues, alongside introducing more realistic evaluation metrics and settings. This comprehensive approach aims to provide a more accurate assessment of code LMs' performance in real-world conditions. Evaluating code LMs on PrimeVul reveals that existing benchmarks significantly overestimate the performance of these models. For instance, a state-of-the-art 7B model scored 68.26% F1 on BigVul but only 3.09% F1 on PrimeVul. Attempts to improve performance through advanced training techniques and larger models like GPT-3.5 and GPT-4 were unsuccessful, with results akin to random guessing in the most stringent settings. These findings underscore the considerable gap between current capabilities and the practical requirements for deploying code LMs in security roles, highlighting the need for more innovative research in this domain.","url_abs":"https://arxiv.org/abs/2403.18624v2","url_pdf":"https://arxiv.org/pdf/2403.18624v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vulnerability-detection-with-code-language","repo_url":"https://github.com/dlvuldet/primevul","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"vulnerability-detection","task_name":"Vulnerability Detection"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"set","method_name":"SET"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.18624","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.18624"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dlvuldet/primevul","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"unverified":4},"by_repo_kind":{"official":{"samples":10,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7559f2b5dd244067","entry":"calculate_metrics","repo":"dlvuldet/primevul","repo_kind":"official","path":"os_expr/run_ft.py","file_url":"https://github.com/dlvuldet/primevul/blob/HEAD/os_expr/run_ft.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7559f2b5dd244067"}},{"code_sha256_prefix":"879ec3454c9a7395","entry":"calculate_vul_det_score","repo":"dlvuldet/primevul","repo_kind":"official","path":"calc_vd_score.py","file_url":"https://github.com/dlvuldet/primevul/blob/HEAD/calc_vd_score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"879ec3454c9a7395"}},{"code_sha256_prefix":"630c44e7fdc892f8","entry":"construct_prompts","repo":"dlvuldet/primevul","repo_kind":"official","path":"openai_expr/run_prompting.py","file_url":"https://github.com/dlvuldet/primevul/blob/HEAD/openai_expr/run_prompting.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"630c44e7fdc892f8"}},{"code_sha256_prefix":"81aa84eb56e35382","entry":"convert_examples_to_features","repo":"dlvuldet/primevul","repo_kind":"official","path":"os_expr/run_ft.py","file_url":"https://github.com/dlvuldet/primevul/blob/HEAD/os_expr/run_ft.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81aa84eb56e35382"}},{"code_sha256_prefix":"b17fb0860dac6e1d","entry":"convert_examples_to_features","repo":"dlvuldet/primevul","repo_kind":"official","path":"os_expr/run_ft_accelerator.py","file_url":"https://github.com/dlvuldet/primevul/blob/HEAD/os_expr/run_ft_accelerator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b17fb0860dac6e1d"}},{"code_sha256_prefix":"62a2fd5c0079c379","entry":"convert_examples_to_features","repo":"dlvuldet/primevul","repo_kind":"official","path":"os_expr/run_ft_clr.py","file_url":"https://github.com/dlvuldet/primevul/blob/HEAD/os_expr/run_ft_clr.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"62a2fd5c0079c379"}},{"code_sha256_prefix":"5737bb1a8eda93a3","entry":"evaluate","repo":"dlvuldet/primevul","repo_kind":"official","path":"os_expr/run_ft.py","file_url":"https://github.com/dlvuldet/primevul/blob/HEAD/os_expr/run_ft.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5737bb1a8eda93a3"}},{"code_sha256_prefix":"24722c9f04658f98","entry":"evaluate","repo":"dlvuldet/primevul","repo_kind":"official","path":"os_expr/run_ft_accelerator.py","file_url":"https://github.com/dlvuldet/primevul/blob/HEAD/os_expr/run_ft_accelerator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"24722c9f04658f98"}},{"code_sha256_prefix":"ad6ec6651bb9bdf0","entry":"evaluate","repo":"dlvuldet/primevul","repo_kind":"official","path":"os_expr/run_ft_clr.py","file_url":"https://github.com/dlvuldet/primevul/blob/HEAD/os_expr/run_ft_clr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad6ec6651bb9bdf0"}},{"code_sha256_prefix":"b5884da68dade4d3","entry":"truncate_tokens_from_messages","repo":"dlvuldet/primevul","repo_kind":"official","path":"openai_expr/run_prompting.py","file_url":"https://github.com/dlvuldet/primevul/blob/HEAD/openai_expr/run_prompting.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b5884da68dade4d3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}