{"url":"/dataset/securityeval","name":"SecurityEval","full_name":null,"description_markdown":"Automated source code generation is currently a popular machine learning-based task. It can be helpful for software developers to write functionally correct code from a given context. However, just like human developers, a code generation model can produce vulnerable code, which the developers can mistakenly use. For this reason, evaluating the security of a code generation model is a must. In this paper, we describe SecurityEval, an evaluation dataset to fulfill this purpose. It contains 130 samples for 75 vulnerability types, which are mapped to the Common Weakness Enumeration (CWE). We also demonstrate using our dataset to evaluate one open-source (i.e., InCoder) and one closed-source code generation model (i.e., GitHub Copilot).","description_withheld":null,"homepage":"https://github.com/s2e-lab/SecurityEval","introduced_date":"2022-11-09","introduced_date_note":null,"introduced_by":{"paper":"/paper/securityeval-dataset-mining-vulnerability","title":"SecurityEval Dataset: Mining Vulnerability Examples to Evaluate Machine Learning-Based Code Generation Techniques","first_author":"Mohammed Latif Siddiq","url":null},"license":null,"modalities":[],"tasks":[{"name":"Code Generation","url":"/task/code-generation","datasets_with_task":"/datasets/task/code-generation"}],"languages":[{"name":"English","url":"/datasets/language/english"}],"variants":["SecurityEval"],"data_loaders":[{"repo":"https://github.com/s2e-lab/SecurityEval","url":"https://github.com/s2e-lab/SecurityEval","frameworks":[]}],"num_papers_in_archive":17,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[],"papers_with_a_benchmark_row":[],"syntology_totals":{"read_at":"2026-09-24T18:15:14+00:00","papers_with_samples":0,"samples_harvested":0,"samples_ran":0,"samples_unverified":0,"pointer_only_for_licence":0,"papers_with_no_sample_that_ran":0,"note":"the per-paper counts above, summed; not a rate"},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks."}