Datasets › VerilogEval

VerilogEval

28 Oct 2023 archive 2025-07-28

VerilogEval Dataset

The VerilogEval Dataset is a benchmark specifically designed to assess the ability of large language models (LLMs) to generate syntactically correct and functionally accurate Verilog code. Introduced in the paper VerilogEval: Evaluating Large Language Models for Verilog Code Generation, it has become a cornerstone for research in hardware code generation.


Dataset Characteristics

  • Diverse Problem Set:
    The dataset comprises 156 distinct problems sourced from HDLBits, covering a wide range of digital design tasks—from simple combinational circuits to more complex sequential and state-based designs.

  • Dual Descriptions:
    It includes two types of problem statements:

  • VerilogEval-human: Handcrafted descriptions by experts, closely reflecting real-world design challenges.
  • VerilogEval-machine: Machine-generated descriptions, typically more verbose, to simulate automated problem formulation.

  • Automated Evaluation Framework:
    Each problem comes with a canonical solution and an evaluation harness that uses simulation (e.g., via Icarus Verilog) to verify the functional correctness of generated code. Evaluation metrics such as pass@k account for the non-deterministic outputs of LLMs.

  • Rich Evaluation Metrics:
    The dataset supports detailed error analysis by classifying failures (e.g., syntax errors, simulation mismatches) and quantifying performance improvements through metrics like pass@1 and pass@5.


Motivations and Content Summary

The primary motivations behind the VerilogEval Dataset are:

  • Standardized Benchmarking:
    To provide a reproducible framework for evaluating and comparing the performance of various LLMs on Verilog code generation tasks.

  • Advancing Hardware Design Automation:
    By focusing on Verilog—a key hardware description language—the dataset encourages research that bridges AI and digital hardware design, ultimately helping to automate and accelerate chip design processes.

  • Facilitating Model Improvement:
    The dataset’s detailed error classifications and pass rate metrics help pinpoint specific weaknesses in LLM-generated code, guiding future research in prompt engineering, in-context learning, and fine-tuning methods for improved performance.


Potential Use Cases

  • Benchmarking and Comparison:
    Researchers can use the dataset to measure and compare the performance of different LLMs (e.g., GPT-4, CodeGen, etc.) in generating correct Verilog code.

  • Prompt Engineering Research:
    The dataset allows exploration of the effects of prompt tuning and in-context learning on the quality of generated hardware description language code.

  • Fine-Tuning and Domain Adaptation:
    It serves as an excellent resource for supervised fine-tuning, enabling models to adapt better to the nuances of Verilog and hardware design tasks.

  • Educational Resource:
    Educators and students can leverage the dataset to practice Verilog coding, test design understanding, and learn automated testing techniques in digital design courses.

  • EDA Tool Integration:
    The dataset can be integrated into Electronic Design Automation (EDA) workflows for automatic code verification, debugging, and performance analysis.


Example Evaluation Metric

An example of a metric used in the evaluation is the pass rate, defined as:

Pass Rate = (Number of samples that pass functional tests)/n

This metric quantifies the reliability of generated code by considering the non-deterministic nature of LLM outputs, where multiple samples are generated per problem.

Benchmarks archive 2025-07-28

All 1 leaderboard whose dataset resolves to this page shown (sort by any header). "First row" is the archive's own first row at snapshot, in the archive's row order; nothing here re-ranks and metric direction is not asserted.

First row (archive order)PaperCode
Code Generation VerilogEval Nexus (Claude 3.5 Sonnet) Pass Rate 85.9 Nexus: A Lightweight and Scalable Multi-Agent Framework... PrimisAI/nexus 1 Compare

Papers archive 2025-07-28

papers with a benchmark row: 1 · with a code link: 1 · where Syntology ran a sample: 0 (0 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument) Syntology

Show: all papers with a benchmark rowonly where code ran: none of the 1 papers with a benchmark row has a Syntology run on record

Page 1 of 1: paper 1 of 1 with a leaderboard row on this dataset's benchmarks, newest first by the archive's date (ties by slug; undated papers last).

The archive's own "papers using this dataset" list was never published, so this is the benchmark-backed subset, not that list; the archive's count for this dataset is 2. The Syntology column is from Syntology's graph, stated per sample; it is not part of any archive number. A line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified” (C is a part of N, never taken away from it); the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. When the archive marks a repository official for the paper, the cell starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record.

PaperCodeResultsDateSamples run Syntology
Nexus: A Lightweight and Scalable Multi-Agent Framework for Complex Tasks Automation 1 1 26 Feb 2025 not harvested

Dataset loaders archive 2025-07-28

No loader listed in the archive.

Tasks archive 2025-07-28

License archive 2025-07-28

MIT License

Modalities archive 2025-07-28

Languages archive 2025-07-28

Variants archive 2025-07-28

  • VerilogEval

1 variant name, as the archive lists them.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections