{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ecco-can-we-improve-model-generated-code","title":"ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?","arxiv_id":"2407.14044","date":"2024-07-19","proceeding":null,"authors":["Siddhant Waghjale","Vishruth Veerendranath","Zora Zhiruo Wang","Daniel Fried"],"abstract":"Although large language models (LLMs) have been largely successful in generating functionally correct programs, conditioning models to produce efficient solutions while ensuring correctness remains a challenge. Further, unreliability in benchmarking code efficiency is a hurdle across varying hardware specifications for popular interpreted languages such as Python. In this paper, we present ECCO, a reproducible benchmark for evaluating program efficiency via two paradigms: natural language (NL) based code generation and history-based code editing. On ECCO, we adapt and thoroughly investigate the three most promising existing LLM-based approaches: in-context learning, iterative refinement with execution or NL feedback, and fine-tuning conditioned on execution and editing history. While most methods degrade functional correctness and moderately increase program efficiency, we find that adding execution information often helps maintain functional correctness, and NL feedback enhances more on efficiency. We release our benchmark to support future work on LLM-based generation of efficient code.","url_abs":"https://arxiv.org/abs/2407.14044v2","url_pdf":"https://arxiv.org/pdf/2407.14044v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ecco-can-we-improve-model-generated-code","repo_url":"https://github.com/codeeff/ecco","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"ecco-can-we-improve-model-generated-code","repo_url":"https://github.com/judge0/judge0","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"GPL-3.0"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.14044","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.14044"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/huggingface/transformers","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/microsoft/DeepSpeed","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/codeeff/ecco","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/judge0/judge0","reach":{"status":"ok","spdx":"GPL-3.0"}}],"summary":{"ran":10,"unverified":4},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1},"found_in_text":{"samples":9,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"bc9117fa9c34b5ef","entry":"apply_scores_before_experts_if_enabled","repo":"microsoft/DeepSpeed","repo_kind":"found_in_text","path":"deepspeed/module_inject/auto_ep_layer.py","file_url":"https://github.com/microsoft/DeepSpeed/blob/HEAD/deepspeed/module_inject/auto_ep_layer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bc9117fa9c34b5ef"}},{"code_sha256_prefix":"30700c5fc6cd1064","entry":"autoep_folding_gradient_reduction_strategy","repo":"microsoft/DeepSpeed","repo_kind":"found_in_text","path":"deepspeed/module_inject/auto_ep_folding.py","file_url":"https://github.com/microsoft/DeepSpeed/blob/HEAD/deepspeed/module_inject/auto_ep_folding.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"30700c5fc6cd1064"}},{"code_sha256_prefix":"ffe9c8c7bc502548","entry":"build_coder_prompts","repo":"codeeff/ecco","repo_kind":"official","path":"experiments/utils.py","file_url":"https://github.com/codeeff/ecco/blob/HEAD/experiments/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ffe9c8c7bc502548"}},{"code_sha256_prefix":"882b572136753409","entry":"build_feedback_prompts","repo":"codeeff/ecco","repo_kind":"official","path":"experiments/utils.py","file_url":"https://github.com/codeeff/ecco/blob/HEAD/experiments/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"882b572136753409"}},{"code_sha256_prefix":"ad8cfe379b811c40","entry":"expected_folding_group_tables","repo":"microsoft/DeepSpeed","repo_kind":"found_in_text","path":"deepspeed/module_inject/auto_ep_folding.py","file_url":"https://github.com/microsoft/DeepSpeed/blob/HEAD/deepspeed/module_inject/auto_ep_folding.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ad8cfe379b811c40"}},{"code_sha256_prefix":"3e1be7ab52fa3b07","entry":"get_execution_feedback","repo":"codeeff/ecco","repo_kind":"official","path":"experiments/utils.py","file_url":"https://github.com/codeeff/ecco/blob/HEAD/experiments/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3e1be7ab52fa3b07"}},{"code_sha256_prefix":"66a3320fa7fe659e","entry":"human_readable_size","repo":"microsoft/DeepSpeed","repo_kind":"found_in_text","path":"deepspeed/env_report.py","file_url":"https://github.com/microsoft/DeepSpeed/blob/HEAD/deepspeed/env_report.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"66a3320fa7fe659e"}},{"code_sha256_prefix":"5f8a25a43a552f16","entry":"local_folding_ranks","repo":"microsoft/DeepSpeed","repo_kind":"found_in_text","path":"deepspeed/module_inject/auto_ep_folding.py","file_url":"https://github.com/microsoft/DeepSpeed/blob/HEAD/deepspeed/module_inject/auto_ep_folding.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5f8a25a43a552f16"}},{"code_sha256_prefix":"7a2e41bee4f05c5a","entry":"move","repo":"microsoft/DeepSpeed","repo_kind":"found_in_text","path":"deepspeed/module_inject/auto_tp.py","file_url":"https://github.com/microsoft/DeepSpeed/blob/HEAD/deepspeed/module_inject/auto_tp.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7a2e41bee4f05c5a"}},{"code_sha256_prefix":"a2da1bc94fa8af7d","entry":"resolve_combine_impl","repo":"microsoft/DeepSpeed","repo_kind":"found_in_text","path":"deepspeed/module_inject/auto_ep_layer.py","file_url":"https://github.com/microsoft/DeepSpeed/blob/HEAD/deepspeed/module_inject/auto_ep_layer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a2da1bc94fa8af7d"}},{"code_sha256_prefix":"45604a14e400b2f3","entry":"get_annotations","repo":"microsoft/DeepSpeed","repo_kind":"found_in_text","path":"deepspeed/compat.py","file_url":"https://github.com/microsoft/DeepSpeed/blob/HEAD/deepspeed/compat.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"45604a14e400b2f3"}},{"code_sha256_prefix":"8c9ef2f97041c68b","entry":"get_annotations_from_namespace","repo":"microsoft/DeepSpeed","repo_kind":"found_in_text","path":"deepspeed/compat.py","file_url":"https://github.com/microsoft/DeepSpeed/blob/HEAD/deepspeed/compat.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8c9ef2f97041c68b"}},{"code_sha256_prefix":"3f906c29f0acb355","entry":"judge_eval_single_test","repo":"codeeff/ecco","repo_kind":"official","path":"evaluation/utils.py","file_url":"https://github.com/codeeff/ecco/blob/HEAD/evaluation/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3f906c29f0acb355"}},{"code_sha256_prefix":"10835a1ccca19d51","entry":"judge_submit","repo":"codeeff/ecco","repo_kind":"official","path":"evaluation/utils.py","file_url":"https://github.com/codeeff/ecco/blob/HEAD/evaluation/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"10835a1ccca19d51"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}