{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/codeeditorbench-evaluating-code-editing","title":"CodeEditorBench: Evaluating Code Editing Capability of Large Language Models","arxiv_id":"2404.03543","date":"2024-04-04","proceeding":null,"authors":["Jiawei Guo","Ziming Li","Xueling Liu","Kaijing Ma","Tianyu Zheng","Zhouliang Yu","Ding Pan","Yizhi Li","Ruibo Liu","Yue Wang","Shuyue Guo","Xingwei Qu","Xiang Yue","Ge Zhang","Wenhu Chen","Jie Fu"],"abstract":"Large Language Models (LLMs) for code are rapidly evolving, with code editing emerging as a critical capability. We introduce CodeEditorBench, an evaluation framework designed to rigorously assess the performance of LLMs in code editing tasks, including debugging, translating, polishing, and requirement switching. Unlike existing benchmarks focusing solely on code generation, CodeEditorBench emphasizes real-world scenarios and practical aspects of software development. We curate diverse coding challenges and scenarios from five sources, covering various programming languages, complexity levels, and editing tasks. Evaluation of 19 LLMs reveals that closed-source models (particularly Gemini-Ultra and GPT-4), outperform open-source models in CodeEditorBench, highlighting differences in model performance based on problem types and prompt sensitivities. CodeEditorBench aims to catalyze advancements in LLMs by providing a robust platform for assessing code editing capabilities. We will release all prompts and datasets to enable the community to expand the dataset and benchmark emerging LLMs. By introducing CodeEditorBench, we contribute to the advancement of LLMs in code editing and provide a valuable resource for researchers and practitioners.","url_abs":"https://arxiv.org/abs/2404.03543v3","url_pdf":"https://arxiv.org/pdf/2404.03543v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"codeeditorbench-evaluating-code-editing","repo_url":"https://github.com/CodeEditorBench/CodeEditorBench","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.03543","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.03543"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/CodeEditorBench/CodeEditorBench","reach":null}],"summary":{"ran_draft_wrong":4},"by_repo_kind":{"listed":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"081d7101e7078625","entry":"evaluate","repo":"CodeEditorBench/CodeEditorBench","repo_kind":"listed","path":"vllm_inference.py","file_url":"https://github.com/CodeEditorBench/CodeEditorBench/blob/HEAD/vllm_inference.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"081d7101e7078625"}},{"code_sha256_prefix":"7f95178793c0b98c","entry":"extract_from_first_keyword","repo":"CodeEditorBench/CodeEditorBench","repo_kind":"listed","path":"result_postprocess.py","file_url":"https://github.com/CodeEditorBench/CodeEditorBench/blob/HEAD/result_postprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7f95178793c0b98c"}},{"code_sha256_prefix":"95cb3a7ff8f342c6","entry":"preprocess","repo":"CodeEditorBench/CodeEditorBench","repo_kind":"listed","path":"result_postprocess.py","file_url":"https://github.com/CodeEditorBench/CodeEditorBench/blob/HEAD/result_postprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"95cb3a7ff8f342c6"}},{"code_sha256_prefix":"fffc4ecc055a301a","entry":"read_jsonl_file","repo":"CodeEditorBench/CodeEditorBench","repo_kind":"listed","path":"result_postprocess.py","file_url":"https://github.com/CodeEditorBench/CodeEditorBench/blob/HEAD/result_postprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fffc4ecc055a301a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}