Datasets › EnvBench
EnvBench
EnvBench is a comprehensive benchmark for automating environment setup - an important task in software engineering. We have collected the largest dataset to date for this task and introduced a robust framework for developing and evaluating LLM-based agents that tackle environment setup challenges.
Our benchmark includes:
- 994 repositories: 329 Python and 665 JVM-based (Java, Kotlin) projects
- Genuine configuration challenges: Carefully selected repositories that cannot be configured with simple deterministic scripts
- Evaluation metrics: Static analysis for missing imports in Python and compilation checks for JVM languages
- Baselines: Zero-shot baselines and agentic workflows tested with GPT-4o and GPT-4o-mini
Benchmarks archive 2025-07-28
No leaderboard in the archive resolves to this dataset.
Papers archive 2025-07-28
No paper in the archive has a leaderboard row on this dataset; the archive counts 1 paper for it but never published that list.
Dataset loaders archive 2025-07-28
No loader listed in the archive.
Tasks archive 2025-07-28
No task tagged in the archive.
License archive 2025-07-28
No licence recorded in the archive. Absence here is not a statement about the dataset's terms.
Modalities archive 2025-07-28
No modality tagged.
Languages archive 2025-07-28
No language tagged.
Variants archive 2025-07-28
- EnvBench
1 variant name, as the archive lists them.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections