{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-static-evaluation-of-code-completion-by","title":"A Static Evaluation of Code Completion by Large Language Models","arxiv_id":"2306.03203","date":"2023-06-05","proceeding":null,"authors":["Hantian Ding","Varun Kumar","Yuchen Tian","Zijian Wang","Rob Kwiatkowski","Xiaopeng Li","Murali Krishna Ramanathan","Baishakhi Ray","Parminder Bhatia","Sudipta Sengupta","Dan Roth","Bing Xiang"],"abstract":"Large language models trained on code have shown great potential to increase productivity of software developers. Several execution-based benchmarks have been proposed to evaluate functional correctness of model-generated code on simple programming problems. Nevertheless, it is expensive to perform the same evaluation on complex real-world projects considering the execution cost. On the contrary, static analysis tools such as linters, which can detect errors without running the program, haven't been well explored for evaluating code generation models. In this work, we propose a static evaluation framework to quantify static errors in Python code completions, by leveraging Abstract Syntax Trees. Compared with execution-based evaluation, our method is not only more efficient, but also applicable to code in the wild. For experiments, we collect code context from open source repos to generate one million function bodies using public models. Our static analysis reveals that Undefined Name and Unused Variable are the most common errors among others made by language models. Through extensive studies, we also show the impact of sampling temperature, model size, and context on static errors in code completions.","url_abs":"https://arxiv.org/abs/2306.03203v1","url_pdf":"https://arxiv.org/pdf/2306.03203v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"code-completion","task_name":"Code Completion"},{"task_slug":"code-generation","task_name":"Code Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.03203","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.03203"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/PyCQA/pyflakes","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":4},"by_repo_kind":{"found_in_text":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"109b85aa8d938931","entry":"getAlternatives","repo":"PyCQA/pyflakes","repo_kind":"found_in_text","path":"pyflakes/checker.py","file_url":"https://github.com/PyCQA/pyflakes/blob/HEAD/pyflakes/checker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"109b85aa8d938931"}},{"code_sha256_prefix":"7734738799f6eb2f","entry":"isPythonFile","repo":"PyCQA/pyflakes","repo_kind":"found_in_text","path":"pyflakes/api.py","file_url":"https://github.com/PyCQA/pyflakes/blob/HEAD/pyflakes/api.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7734738799f6eb2f"}},{"code_sha256_prefix":"e1fac60c214ab3fa","entry":"iter_dict_children","repo":"PyCQA/pyflakes","repo_kind":"found_in_text","path":"pyflakes/checker.py","file_url":"https://github.com/PyCQA/pyflakes/blob/HEAD/pyflakes/checker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e1fac60c214ab3fa"}},{"code_sha256_prefix":"99718e5044230670","entry":"parse_percent_format","repo":"PyCQA/pyflakes","repo_kind":"found_in_text","path":"pyflakes/checker.py","file_url":"https://github.com/PyCQA/pyflakes/blob/HEAD/pyflakes/checker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"99718e5044230670"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}