NaturalCodeBench

NaturalCodeBench (NCB) is a comprehensive code benchmark designed to mirror the complexity and variety of scenarios in real coding tasks¹². It comprises 402 high-quality problems in Python and Java, meticulously selected from an online coding service, covering 6 different domains¹².

The seed problems of NCB are cleaned from the queries in coding online services, spanning across 6 domains: Artificial Intelligence, Data Science, Algorithm and Data Structure, Front-End, Software Engineering, and System Administration¹.

Here is a summary of the number of problems in each domain¹: - Software Engineering: 132 problems - Data Science: 100 problems - Algorithm and Data Structure: 95 problems - System Administration: 33 problems - Artificial Intelligence: 28 problems - Front-End: 14 problems

The development set of NCB, which contains 140 problems (70 in Python and 70 in Java), is released for research purposes¹. The data format includes a unique identifier for each question, the problem description, testcases, setup code, a reference solution, and the domain of the problem¹.

(1) GitHub - THUDM/NaturalCodeBench. https://github.com/THUDM/NaturalCodeBench. (2) NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval .... https://arxiv.org/abs/2405.04520. (3) 清华、智谱AI 团队推出代码评测基准 NaturalCodeBench .... https://blog.csdn.net/www3300300/article/details/138752139. (4) NaturalCodeBench: Examining Coding Performance .... https://www.x-mol.com/paper/1788507497390862336. (5) undefined. https://doi.org/10.48550/arXiv.2405.04520.