{"url":"/dataset/verilogeval","name":"VerilogEval","full_name":null,"description_markdown":"# VerilogEval Dataset\r\n\r\nThe **VerilogEval Dataset** is a benchmark specifically designed to assess the ability of large language models (LLMs) to generate syntactically correct and functionally accurate Verilog code. Introduced in the paper *VerilogEval: Evaluating Large Language Models for Verilog Code Generation*, it has become a cornerstone for research in hardware code generation.\r\n\r\n---\r\n\r\n## Dataset Characteristics\r\n\r\n- **Diverse Problem Set:**  \r\n  The dataset comprises 156 distinct problems sourced from HDLBits, covering a wide range of digital design tasks—from simple combinational circuits to more complex sequential and state-based designs.\r\n\r\n- **Dual Descriptions:**  \r\n  It includes two types of problem statements:\r\n  - **VerilogEval-human:** Handcrafted descriptions by experts, closely reflecting real-world design challenges.\r\n  - **VerilogEval-machine:** Machine-generated descriptions, typically more verbose, to simulate automated problem formulation.\r\n\r\n- **Automated Evaluation Framework:**  \r\n  Each problem comes with a canonical solution and an evaluation harness that uses simulation (e.g., via Icarus Verilog) to verify the functional correctness of generated code. Evaluation metrics such as pass@k account for the non-deterministic outputs of LLMs.\r\n\r\n- **Rich Evaluation Metrics:**  \r\n  The dataset supports detailed error analysis by classifying failures (e.g., syntax errors, simulation mismatches) and quantifying performance improvements through metrics like pass@1 and pass@5.\r\n\r\n---\r\n\r\n## Motivations and Content Summary\r\n\r\nThe primary motivations behind the VerilogEval Dataset are:\r\n\r\n- **Standardized Benchmarking:**  \r\n  To provide a reproducible framework for evaluating and comparing the performance of various LLMs on Verilog code generation tasks.\r\n\r\n- **Advancing Hardware Design Automation:**  \r\n  By focusing on Verilog—a key hardware description language—the dataset encourages research that bridges AI and digital hardware design, ultimately helping to automate and accelerate chip design processes.\r\n\r\n- **Facilitating Model Improvement:**  \r\n  The dataset’s detailed error classifications and pass rate metrics help pinpoint specific weaknesses in LLM-generated code, guiding future research in prompt engineering, in-context learning, and fine-tuning methods for improved performance.\r\n\r\n---\r\n\r\n## Potential Use Cases\r\n\r\n- **Benchmarking and Comparison:**  \r\n  Researchers can use the dataset to measure and compare the performance of different LLMs (e.g., GPT-4, CodeGen, etc.) in generating correct Verilog code.\r\n\r\n- **Prompt Engineering Research:**  \r\n  The dataset allows exploration of the effects of prompt tuning and in-context learning on the quality of generated hardware description language code.\r\n\r\n- **Fine-Tuning and Domain Adaptation:**  \r\n  It serves as an excellent resource for supervised fine-tuning, enabling models to adapt better to the nuances of Verilog and hardware design tasks.\r\n\r\n- **Educational Resource:**  \r\n  Educators and students can leverage the dataset to practice Verilog coding, test design understanding, and learn automated testing techniques in digital design courses.\r\n\r\n- **EDA Tool Integration:**  \r\n  The dataset can be integrated into Electronic Design Automation (EDA) workflows for automatic code verification, debugging, and performance analysis.\r\n\r\n---\r\n\r\n## Example Evaluation Metric\r\n\r\nAn example of a metric used in the evaluation is the **pass rate**, defined as:\r\n\r\n\\[\r\n\\text{Pass Rate} = \\frac{\\text{Number of samples that pass functional tests}}{n}\r\n\\]\r\n\r\nThis metric quantifies the reliability of generated code by considering the non-deterministic nature of LLM outputs, where multiple samples are generated per problem.","description_withheld":null,"homepage":"https://github.com/nvlabs/verilog-eval","introduced_date":"2023-10-28","introduced_date_note":null,"introduced_by":null,"license":{"name":"MIT License","url":"https://github.com/NVlabs/verilog-eval/blob/main/LICENSE"},"modalities":[{"name":"Texts","url":"/datasets/modality/texts"}],"tasks":[{"name":"Code Generation","url":"/task/code-generation","datasets_with_task":"/datasets/task/code-generation"}],"languages":[{"name":"English","url":"/datasets/language/english"}],"variants":["VerilogEval"],"data_loaders":[],"num_papers_in_archive":2,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[{"leaderboard":"/sota/code-generation-on-verilogeval","task":"Code Generation","dataset_variant":"VerilogEval","rows":1,"metrics":["Pass Rate"],"first_row_in_archive_order":{"model":"Nexus (Claude 3.5 Sonnet)","paper":"/paper/nexus-a-lightweight-and-scalable-multi-agent","metrics":{"Pass Rate":"85.9"},"code_links":[{"title":"PrimisAI/nexus","url":"https://github.com/PrimisAI/nexus"}]},"note":"rows are the archive's own order at snapshot; nothing here re-ranks them"}],"papers_with_a_benchmark_row":[{"paper":"/paper/nexus-a-lightweight-and-scalable-multi-agent","title":"Nexus: A Lightweight and Scalable Multi-Agent Framework for Complex Tasks Automation","date":"2025-02-26","rows_on_this_dataset":1,"code_links":1,"syntology":null}],"syntology_totals":{"read_at":"2026-09-24T18:15:14+00:00","papers_with_samples":0,"samples_harvested":0,"samples_ran":0,"samples_unverified":0,"pointer_only_for_licence":0,"papers_with_no_sample_that_ran":0,"note":"the per-paper counts above, summed; not a rate"},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks."}