RepoQA

RepoQA is a benchmark that aims to exercise the long-context code understanding ability of Language Learning Models (LLMs)². It supports repositories from 5 programming languages: Python, C++, TypeScript, Rust, and Java².

The goal of RepoQA is to evaluate LLMs on long-context tasks that can reflect real-life uses². For example, it includes a function called repoqa.search_needle_function that evaluates model outputs and computes scores after text generation³.

(1) Releases · evalplus/repoqa · GitHub. https://github.com/evalplus/repoqa/releases. (2) RepoQA: Evaluating Long-Context Code Understanding - GitHub. https://github.com/evalplus/repoqa. (3) BrowserBench.org — Browser Benchmarks. https://browserbench.org/.