{"url":"/dataset/coffe","name":"COFFE","full_name":"COFFE: A Code Efficiency Benchmark for Code Generation","description_markdown":"# COFFE\r\n\r\nCOFFE is a Python benchmark for evaluating the time efficiency of LLM-generated code. It is released by the FSE'25 paper \"[COFFE: A Code Efficiency Benchmark for Code Generation](https://arxiv.org/abs/2502.02827)\". You can also refer to the [project webpage](https://www.yunpeng.site/projects/coffe/) for more details.\r\n\r\n## Data\r\n\r\nCOFFE is designed for evaluating both function-level code and file-level code. It contains selected instances from HumanEval, MBPP, APPS and Code Contests. COFFE keeps the original test cases in these benchmarks as *correctness test cases* and adds new test cases designed for time efficiency evaluation as *stressful test cases*.\r\n\r\n**Statistics:**\r\n\r\n|Category|#Instance|#Solution/Instance|#Correctness/Instance | #Stressful/Instance|\r\n|----|----|----|----|----|\r\n|Function-Level|398|1.00|5.72|4.99|\r\n|File-Level|358|66.93|43.68|4.95|\r\n\r\n**Data Files:**\r\n\r\nAll instances in COFFE are in `Coffe/datasets`, where `Coffe/datasets/function` contains all function-level instances and `Coffe/datasets/file` contains all file-level instances. In each repo:\r\n- `best_solutions.json` contains the best ground truth solution COFFE uses to calculate efficient@1 and speedup.\r\n- `stressful_testcases.json` contains all stressful test cases COFFE adds.\r\n- `solutions.json` contains all ground truth solutions in original benchmarks.\r\n- `testcases.json` contains all correctness test cases in original benchmarks.\r\n\r\n## Installation\r\n\r\n**Requirements:**\r\n- Linux Machine\r\n- Docker \r\n- Python>=3.10\r\n\r\nWe suggest you create a virtural environment before installing COFFE.\r\n\r\n1. To use COFFE, please clone this repo in your current workspace `workspace/` and execute:\r\n```bash\r\ncd Coffe && pip install .\r\n```\r\n2. COFFE comes with a docker image, to install it:\r\n```bash\r\ndocker build . -t coffe\r\n```\r\nNote that if your network requires proxy, please modify the Dockerfile in `Coffe/` to indicate it, otherwise the docker image building process could fail.\r\n\r\n3. Go back to your workspace and initialize COFFE:\r\n```bash\r\ncd .. && coffe init\r\n```\r\nIf your installation succeeds, you could see the statistics of COFFE. \r\n\r\n## Usage\r\n\r\n### Pipeline\r\n\r\nWhen you prepare the predictions from LLMs, COFFE provides a pipeline to calculate the efficient@1 and speedup defined in the paper:\r\n```bash\r\ncoffe pipe <benchmark_name> <output_repo> \r\n-p <pred_file_1>,...,<pred_file_n> \r\n-f <efficient_at_1/speedup> \r\n-n <number_of_workers>\r\n```\r\nThis command has four phases:\r\n1. santize the predictions.\r\n2. select the correct predictions based on correctness test cases.\r\n3. evaluate the GPU instruction count based on stressful test cases.\r\n4. calculate the final metrics.\r\n\r\nFor example:\r\n```bash\r\ncoffe pipe function Coffe/examples/function -p Coffe/examples/function/GPT-4o.json -f efficient_at_1 -n 8\r\n```\r\nThis command evaluates the predictions from GPT-4o on the function-level instances of COFFE. If you want to evaluate other LLMs, please prepare a `JSON` file with the same format as `Coffe/examples/function/GPT-4o.json`. \r\n\r\n**Prediction File Format:**\r\n\r\nIn the `JSON` file, the key is the prompt used to query the LLM for the results, you could get the prompts in `datasets/function/prompts.json` and `datasets/file/prompts.json`. The value contains two objects, the first is a list contains the raw outputs from LLMs and the second is an indicator for the whether the raw output is valid.\r\n\r\n**Note:**\r\n\r\nIn default, COFFE will run all predictions in docker. However, if you could not successfully install the docker or want to run the predictions on the host machine, you can add the `-x` option.\r\n\r\n\r\n### Single Evaluation\r\n\r\nThe `pipe` command provides an entire pipeline for calculating the final metrics. This pipeline could also be completed by executing the following four single evaluation commands.\r\n\r\n1. Sanitize the predictions\r\n```bash\r\ncoffe eval <benchmark_name> <output_repo> \r\n-p <pred_file_1>,...,<pred_file_n> \r\n-m compilable_rate\r\n```\r\nThis commands output a file ending with `SOLUTIONS.json` that contains the predictions without syntax errors.\r\n\r\n2. Select correct predictions\r\n```bash\r\ncoffe eval <benchmark_name> <output_repo> \r\n-p <pred_file_1>,...,<pred_file_n> \r\n-m correctness\r\n-n <number_of_workers>\r\n```\r\nThis commands accept prediction files ending with `SOLUTIONS.json` and output a file ending with `PASSED_SOLUTIONS.json` that contains the predictions pass all correctness solutions.\r\n\r\n**Note:**\r\nThis command will combine all correct solutions and ground truth solutions together into files `<dataset_name>_all_indexes.json` (used in step 4) and `<dataset_name>_all_PASSED_SOLUTIONS.json` for the next step.\r\n\r\n3. Evaluate the GPU instruction count\r\n```bash\r\ncoffe eval <benchmark_name> <output_repo> \r\n-p <pred_file>\r\n-m instr_count\r\n-n <number_of_workers>\r\n```\r\nThis command will evaluate the GPU instruction count each prediction consumes and output a file ending with `STRESSFUL_INSTRUCTION.json`. \r\n\r\n**Note:**\r\nThis command could only accept one single prediction file ending with `PASSED_SOLUTIONS.json`.\r\n\r\n4. Calculating the efficient@1/speedup\r\n```bash\r\ncoffe eval <benchmark_name> <output_repo> \r\n-p <index_file>,<pred_file>\r\n-m instr_count\r\n-f <efficient_at_1/speedup>\r\n```\r\nThis command calculate the efficient@1 or speedup.\r\n\r\n**Note:**\r\nThis command requires the index file and the instruction file as COFFE compares the performance of predictions with grouth truth solutions to calculate the metrics.\r\n\r\n## STGen\r\n\r\nFor details about the stressful test case generation approach STGen, please see `stgen/`.\r\n\r\n\r\n## Cite\r\nIf you use COFFE, please cite us:\r\n```\r\n@misc{peng2025coffe,\r\n      title={COFFE: A Code Efficiency Benchmark for Code Generation}, \r\n      author={Yun Peng and Jun Wan and Yichen Li and Xiaoxue Ren},\r\n      year={2025},\r\n      eprint={2502.02827},\r\n      archivePrefix={arXiv},\r\n      primaryClass={cs.SE},\r\n      url={https://arxiv.org/abs/2502.02827}, \r\n}\r\n```","description_withheld":null,"homepage":"https://www.yunpeng.site/projects/coffe","introduced_date":"2025-02-05","introduced_date_note":null,"introduced_by":{"paper":"/paper/coffe-a-code-efficiency-benchmark-for-code","title":"COFFE: A Code Efficiency Benchmark for Code Generation","first_author":null,"url":null},"license":{"name":"Apache License 2.0","url":"https://github.com/JohnnyPeng18/Coffe/blob/main/LICENSE"},"modalities":[{"name":"Texts","url":"/datasets/modality/texts"}],"tasks":[{"name":"Code Generation","url":"/task/code-generation","datasets_with_task":"/datasets/task/code-generation"}],"languages":[{"name":"English","url":"/datasets/language/english"}],"variants":["COFFE"],"data_loaders":[],"num_papers_in_archive":1,"source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28"},"benchmarks":[],"papers_with_a_benchmark_row":[],"syntology_totals":{"read_at":"2026-09-24T18:15:14+00:00","papers_with_samples":0,"samples_harvested":0,"samples_ran":0,"samples_unverified":0,"pointer_only_for_licence":0,"papers_with_no_sample_that_ran":0,"note":"the per-paper counts above, summed; not a rate"},"papers_note":"The archive never published its papers-using-dataset list; these are papers with a leaderboard row on this dataset's benchmarks."}