{"url":"/dataset/repoeval","name":"RepoEval","full_name":null,"description_markdown":"**RepoEval** is a benchmark specifically designed for evaluating **repository-level code auto-completion systems**. While existing benchmarks mainly focus on single-file tasks, RepoEval addresses the assessment gap for more complex, real-world, multi-file programming scenarios. Here are the key details about RepoEval:\r\n\r\n1. **Tasks**:\r\n    - **RepoBench-R (Retrieval)**: Measures the system's ability to retrieve the most relevant code snippets from other files as cross-file context.\r\n    - **RepoBench-C (Code Completion)**: Evaluates the system's capability to predict the next line of code with cross-file and in-file context.\r\n    - **RepoBench-P (Pipeline)**: Handles complex tasks that require a combination of both retrieval and next-line prediction¹.\r\n\r\n2. **Languages Supported**:\r\n    - RepoEval supports both **Python** and **Java**¹.\r\n\r\n3. **Purpose**:\r\n    - RepoEval aims to facilitate a more complete comparison of performance and encourage continuous improvement in auto-completion systems¹.\r\n\r\n4. **Availability**:\r\n    - RepoEval is publicly available for use [here](https://arxiv.org/abs/2306.03091) ¹.\r\n\r\nIn summary, RepoEval provides a comprehensive evaluation framework for assessing the effectiveness of repository-level code auto-completion systems, enabling researchers and developers to enhance code productivity and quality.\r\n\r\n(1) [2306.03091] RepoBench: Benchmarking Repository-Level Code Auto .... https://arxiv.org/abs/2306.03091.\r\n(2) [2303.12570] RepoCoder: Repository-Level Code Completion Through .... https://arxiv.org/abs/2303.12570.\r\n(3) [2306.03091] RepoBench: Benchmarking Repository-Level Code Auto .... https://ar5iv.labs.arxiv.org/html/2306.03091.\r\n(4) GitHub - Leolty/repobench: RepoBench: Benchmarking Repository-Level .... https://github.com/Leolty/repobench.\r\n(5) undefined. https://doi.org/10.48550/arXiv.2306.03091.","description_withheld":null,"homepage":"https://github.com/microsoft/CodeT/tree/main/RepoCoder","introduced_date":"2023-03-22","introduced_date_note":null,"introduced_by":{"paper":"/paper/repocoder-repository-level-code-completion","title":"RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation","first_author":"Fengji Zhang","url":null},"license":null,"modalities":[],"tasks":[],"languages":[],"variants":["RepoEval"],"data_loaders":[],"num_papers_in_archive":18,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[],"papers_with_a_benchmark_row":[],"syntology_totals":{"read_at":"2026-09-24T18:15:14+00:00","papers_with_samples":0,"samples_harvested":0,"samples_ran":0,"samples_unverified":0,"pointer_only_for_licence":0,"papers_with_no_sample_that_ran":0,"note":"the per-paper counts above, summed; not a rate"},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks."}