Vulnerability Detection

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

2 leaderboard tables shown for this task, 2 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

7 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

No subtask under this task in the archive's task tree.

Most implemented papers archive 2025-07-28

30 shown of 72 papers with code (216 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security
  8 Jun 2024 5 repositories listed Syntology ran 5 of 5 samples · 0 unverified · 4 pointer-only (licence)
  Large Language Models (LLMs) are being deployed across various domains today.
• VulDeePecker: A Deep Learning-Based System for Vulnerability Detection
  5 Jan 2018 5 repositories listed Syntology ran 0 of 1 samples · 1 unverified
  Since deep learning is motivated to deal with problems that are very different from the problem of vulnerability detection, we need some guiding principles for applying deep learning to vulnerability detection.
• SySeVR: A Framework for Using Deep Learning to Detect Software Vulnerabilities
  18 Jul 2018 4 repositories listed Syntology ran 1 of 8 samples · 7 unverified · 1 pointer-only (licence)
  Our experiments with 4 software products demonstrate the usefulness of the framework: we detect 15 vulnerabilities that are not reported in the National Vulnerability Database.
• Uncovering the Limits of Machine Learning for Automatic Vulnerability Detection
  28 Jun 2023 3 repositories listed
  Using six ML4VD techniques and two datasets, we find (a) that state-of-the-art models severely overfit to unrelated features for predicting the vulnerabilities in the testing data, (b) that the performance gained by…
• SAFE: Self-Attentive Function Embeddings for Binary Similarity
  13 Nov 2018 3 repositories listed Syntology ran 0 of 4 samples · 4 unverified · 1 pointer-only (licence)
  We report the results from a quantitative and qualitative analysis that show how SAFE provides a noticeable performance improvement with respect to previous solutions.
• Deep Smart Contract Intent Detection
  19 Nov 2022 2 repositories listed
  In recent years, research in software security has concentrated on identifying vulnerabilities in smart contracts to prevent significant losses of crypto assets on blockchains.
• Trex: Learning Execution Semantics from Micro-Traces for Binary Similarity
  16 Dec 2020 2 repositories listed
  We thus train the model to learn execution semantics from the functions' micro-traces, without any manual labeling effort.
• Automated Vulnerability Detection in Source Code Using Deep Representation Learning
  11 Jul 2018 2 repositories listed
  The labeled dataset is available at: https://osf.
• Boosting Vulnerability Detection of LLMs via Curriculum Preference Optimization with Synthetic Reasoning Data
  9 Jun 2025 1 repository listed
  Large language models (LLMs) demonstrate considerable proficiency in numerous coding-related tasks; however, their capabilities in detecting software vulnerabilities remain limited.
• SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysis
  27 May 2025 1 repository listed
  As Large Language Models (LLMs) evolve in understanding and generating code, accurately evaluating their reliability in analyzing source code vulnerabilities becomes increasingly vital.
• CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Execution
  21 May 2025 1 repository listed
  Large Language Model (LLM) agents can automate cybersecurity tasks and can adapt to the evolving cybersecurity landscape without re-engineering.
• Are Sparse Autoencoders Useful for Java Function Bug Detection?
  15 May 2025 1 repository listed
  Software vulnerabilities such as buffer overflows and SQL injections are a major source of security breaches.
• Can You Really Trust Code Copilots? Evaluating Large Language Models from a Code Security Perspective
  15 May 2025 1 repository listed
  In this paper, we first propose CoV-Eval, a multi-task benchmark covering various tasks such as code completion, vulnerability repair, vulnerability detection and classification, for comprehensive evaluation of LLM code…
• Enhancing Large Language Models with Faster Code Preprocessing for Vulnerability Detection
  8 May 2025 1 repository listed
  Our results show a 97.
• Program Semantic Inequivalence Game with Large Language Models
  2 May 2025 1 repository listed
  Finding training examples to teach LLMs to solve these tasks can be challenging.
• The Hitchhiker's Guide to Program Analysis, Part II: Deep Thoughts by LLMs
  16 Apr 2025 1 repository listed
  Static analysis plays a crucial role in software vulnerability detection, yet faces a persistent precision-scalability tradeoff.
• R2Vul: Learning to Reason about Software Vulnerabilities with Reinforcement Learning and Structured Reasoning Distillation
  7 Apr 2025 1 repository listed
  Through RLAIF, R2Vul enables LLMs to produce structured, security-aware reasoning that is actionable and reliable while explicitly learning to distinguish valid assessments from misleading ones.
• Responsible Development of Offensive AI
  3 Apr 2025 1 repository listed
  As AI advances, broader consensus is needed to determine research priorities.
• Reasoning with LLMs for Zero-Shot Vulnerability Detection
  22 Mar 2025 1 repository listed
  Automating software vulnerability detection (SVD) remains a critical challenge in an era of increasingly complex and interdependent software systems.
• CASTLE: Benchmarking Dataset for Static Code Analyzers and LLMs towards CWE Detection
  12 Mar 2025 1 repository listed
  LLMs perform exceptionally well in the CASTLE dataset when identifying vulnerabilities in small code snippets.
• MTVHunter: Smart Contracts Vulnerability Detection Based on Multi-Teacher Knowledge Translation
  24 Feb 2025 1 repository listed
  In this paper, we propose a multi-teacher based bytecode vulnerability detection method, namely \textbf{M}ulti-\textbf{T}eacher \textbf{V}ulnerability \textbf{Hunter} (\textbf{MTVHunter}), which delivers effective…
• How to Select Pre-Trained Code Models for Reuse? A Learning Perspective
  7 Jan 2025 1 repository listed
  Pre-training a language model and then fine-tuning it has shown to be an efficient and effective technique for a wide range of code intelligence tasks, such as code generation, code summarization, and vulnerability…
• Investigating Large Language Models for Code Vulnerability Detection: An Experimental Study
  24 Dec 2024 1 repository listed
  However, the effectiveness of LLMs in detecting code vulnerabilities is largely under-explored.
• Leveraging Generative AI to Enhance Automated Vulnerability Scoring
  7 Dec 2024 1 repository listed
  We conduct several experiments to assess the performance of the proposed method against the state-of-the-art.
• CryptoFormalEval: Integrating LLMs and Formal Verification for Automated Cryptographic Protocol Vulnerability Detection
  20 Nov 2024 1 repository listed
  Cryptographic protocols play a fundamental role in securing modern digital infrastructure, but they are often deployed without prior formal verification.
• Is Function Similarity Over-Engineered? Building a Benchmark
  30 Oct 2024 1 repository listed Syntology ran 1 of 1 samples · 0 unverified
  Binary analysis is a core component of many critical security tasks, including reverse engineering, malware analysis, and vulnerability detection.
• Vulnerability Detection via Topological Analysis of Attention Maps
  4 Oct 2024 1 repository listed
  Recently, deep learning (DL) approaches to vulnerability detection have gained significant traction.
• VulScribeR: Exploring RAG-based Vulnerability Augmentation with LLMs
  7 Aug 2024 1 repository listed
  More specifically, we explore three strategies to augment both single and multi-statement vulnerabilities, with LLMs, namely Mutation, Injection, and Extension.
• EaTVul: ChatGPT-based Evasion Attack Against Software Vulnerability Detection
  27 Jul 2024 1 repository listed
  Recently, deep learning has demonstrated promising results in enhancing the accuracy of vulnerability detection and identifying vulnerabilities in software.
• eyeballvul: a future-proof benchmark for vulnerability detection in the wild
  11 Jul 2024 1 repository listed
  The benchmark consists of a list of revisions in different repositories, each associated with the list of known vulnerabilities present at that revision.

Syntology lines on 5 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.