{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mercury-an-efficiency-benchmark-for-llm-code","title":"Mercury: A Code Efficiency Benchmark for Code Large Language Models","arxiv_id":"2402.07844","date":"2024-02-12","proceeding":null,"authors":["Mingzhe Du","Anh Tuan Luu","Bin Ji","Qian Liu","See-Kiong Ng"],"abstract":"Amidst the recent strides in evaluating Large Language Models for Code (Code LLMs), existing benchmarks have mainly focused on the functional correctness of generated code, neglecting the importance of their computational efficiency. To fill the gap, we present Mercury, the first code efficiency benchmark for Code LLMs. It comprises 1,889 Python tasks, each accompanied by adequate solutions that serve as real-world efficiency baselines, enabling a comprehensive analysis of the runtime distribution. Based on the distribution, we introduce a new metric Beyond, which computes a runtime-percentile-weighted Pass score to reflect functional correctness and code efficiency simultaneously. On Mercury, leading Code LLMs can achieve 65% on Pass, while less than 50% on Beyond. Given that an ideal Beyond score would be aligned with the Pass score, it indicates that while Code LLMs exhibit impressive capabilities in generating functionally correct code, there remains a notable gap in their efficiency. Finally, our empirical experiments reveal that Direct Preference Optimization (DPO) serves as a robust baseline for enhancing code efficiency compared with Supervised Fine Tuning (SFT), which paves a promising avenue for future exploration of efficient code generation. Our code and data are available on GitHub: https://github.com/Elfsong/Mercury.","url_abs":"https://arxiv.org/abs/2402.07844v4","url_pdf":"https://arxiv.org/pdf/2402.07844v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mercury-an-efficiency-benchmark-for-llm-code","repo_url":"https://github.com/elfsong/mercury","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"computational-efficiency","task_name":"Computational Efficiency"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.07844","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.07844"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/elfsong/mercury","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"f0b76e881e34bb03","entry":"formatting_prompts_func","repo":"elfsong/mercury","repo_kind":"official","path":"src/sft_train.py","file_url":"https://github.com/elfsong/mercury/blob/HEAD/src/sft_train.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f0b76e881e34bb03"}},{"code_sha256_prefix":"39eb0010095eb2a9","entry":"prompt_generate","repo":"elfsong/mercury","repo_kind":"official","path":"src/dpo_train.py","file_url":"https://github.com/elfsong/mercury/blob/HEAD/src/dpo_train.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39eb0010095eb2a9"}},{"code_sha256_prefix":"defc40f466ef1977","entry":"prompt_generate","repo":"elfsong/mercury","repo_kind":"official","path":"src/sft_train.py","file_url":"https://github.com/elfsong/mercury/blob/HEAD/src/sft_train.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"defc40f466ef1977"}},{"code_sha256_prefix":"c81df5b5c61de217","entry":"get_code_paired","repo":"elfsong/mercury","repo_kind":"official","path":"src/dpo_train.py","file_url":"https://github.com/elfsong/mercury/blob/HEAD/src/dpo_train.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c81df5b5c61de217"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}