{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/codev-bench-how-do-llms-understand-developer","title":"Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion?","arxiv_id":"2410.01353","date":"2024-10-02","proceeding":null,"authors":["Zhenyu Pan","Rongyu Cao","Yongchang Cao","Yingwei Ma","Binhua Li","Fei Huang","Han Liu","Yongbin Li"],"abstract":"Code completion, a key downstream task in code generation, is one of the most frequent and impactful methods for enhancing developer productivity in software development. As intelligent completion tools evolve, we need a robust evaluation benchmark that enables meaningful comparisons between products and guides future advancements. However, existing benchmarks focus more on coarse-grained tasks without industrial analysis resembling general code generation rather than the real-world scenarios developers encounter. Moreover, these benchmarks often rely on costly and time-consuming human annotation, and the standalone test cases fail to leverage minimal tests for maximum repository-level understanding and code coverage. To address these limitations, we first analyze business data from an industrial code completion tool and redefine the evaluation criteria to better align with the developer's intent and desired completion behavior throughout the coding process. Based on these insights, we introduce Codev-Agent, an agent-based system that automates repository crawling, constructs execution environments, extracts dynamic calling chains from existing unit tests, and generates new test samples to avoid data leakage, ensuring fair and effective comparisons. Using Codev-Agent, we present the Code-Development Benchmark (Codev-Bench), a fine-grained, real-world, repository-level, and developer-centric evaluation framework. Codev-Bench assesses whether a code completion tool can capture a developer's immediate intent and suggest appropriate code across diverse contexts, providing a more realistic benchmark for code completion in modern software development.","url_abs":"https://arxiv.org/abs/2410.01353v3","url_pdf":"https://arxiv.org/pdf/2410.01353v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"codev-bench-how-do-llms-understand-developer","repo_url":"https://github.com/LingmaTongyi/Codev-Bench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"code-completion","task_name":"Code Completion"},{"task_slug":"code-generation","task_name":"Code Generation"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.01353","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.01353"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/LingmaTongyi/Codev-Bench","reach":null}],"summary":{"ran_draft_wrong":4,"ran_violates":2},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"0bb087c979542f85","entry":"filter_method_dict","repo":"LingmaTongyi/Codev-Bench","repo_kind":"official","path":"src/prepare.py","file_url":"https://github.com/LingmaTongyi/Codev-Bench/blob/HEAD/src/prepare.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0bb087c979542f85"}},{"code_sha256_prefix":"9d7324ab205df6a2","entry":"get_tree_structure","repo":"LingmaTongyi/Codev-Bench","repo_kind":"official","path":"src/prepare.py","file_url":"https://github.com/LingmaTongyi/Codev-Bench/blob/HEAD/src/prepare.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9d7324ab205df6a2"}},{"code_sha256_prefix":"fa2aa7edff04b45f","entry":"judge_not_cross_line","repo":"LingmaTongyi/Codev-Bench","repo_kind":"official","path":"src/evaluate.py","file_url":"https://github.com/LingmaTongyi/Codev-Bench/blob/HEAD/src/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fa2aa7edff04b45f"}},{"code_sha256_prefix":"f70e396e2ae87b3b","entry":"judge_right_indent","repo":"LingmaTongyi/Codev-Bench","repo_kind":"official","path":"src/evaluate.py","file_url":"https://github.com/LingmaTongyi/Codev-Bench/blob/HEAD/src/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f70e396e2ae87b3b"}},{"code_sha256_prefix":"8cbd00f229e9b8e5","entry":"run_unit_test_command","repo":"LingmaTongyi/Codev-Bench","repo_kind":"official","path":"src/prepare.py","file_url":"https://github.com/LingmaTongyi/Codev-Bench/blob/HEAD/src/prepare.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8cbd00f229e9b8e5"}},{"code_sha256_prefix":"565db4280f75804c","entry":"timeout","repo":"LingmaTongyi/Codev-Bench","repo_kind":"official","path":"src/evaluate.py","file_url":"https://github.com/LingmaTongyi/Codev-Bench/blob/HEAD/src/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"565db4280f75804c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}