{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/out-of-the-bleu-how-should-we-assess-quality","title":"Out of the BLEU: how should we assess quality of the Code Generation models?","arxiv_id":"2208.03133","date":"2022-08-05","proceeding":null,"authors":["Mikhail Evtikhiev","Egor Bogomolov","Yaroslav Sokolov","Timofey Bryksin"],"abstract":"In recent years, researchers have created and introduced a significant number of various code generation models. As human evaluation of every new model version is unfeasible, the community adopted automatic evaluation metrics such as BLEU to approximate the results of human judgement. These metrics originate from the machine translation domain and it is unclear whether they are applicable for the code generation tasks and how well they agree with the human evaluation on this task. There are also other metrics, CodeBLEU and RUBY, developed to estimate the similarity of code, that take into account the properties of source code. However, for these metrics there are hardly any studies on their agreement with the human evaluation. Despite all that, minimal differences in the metric scores have been used in recent papers to claim superiority of some code generation models over the others. In this paper, we present a study on the applicability of six metrics -- BLEU, ROUGE-L, METEOR, ChrF, CodeBLEU, and RUBY -- for evaluation of code generation models. We conduct a study on two different code generation datasets and use human annotators to assess the quality of all models run on these datasets. The results indicate that for the CoNaLa dataset of Python one-liners, none of the metrics can correctly emulate human judgement on which model is better with >95% certainty if the difference in model scores is less than 5 points. For the HearthStone dataset, which consists of classes of a particular structure, a difference in model scores of at least 2 points is enough to claim the superiority of one model over the other. Our findings suggest that the ChrF metric is a better fit for the evaluation of code generation models than the commonly used BLEU and CodeBLEU. Yet, finding a metric for code generation that closely agrees with humans requires additional work.","url_abs":"https://arxiv.org/abs/2208.03133v2","url_pdf":"https://arxiv.org/pdf/2208.03133v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"out-of-the-bleu-how-should-we-assess-quality","repo_url":"https://github.com/JetBrains-Research/codegen-metrics","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"machine-translation","task_name":"Machine Translation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2208.03133","atlas_url":"https://app.syntology.ai/?focus=2208.03133","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2208.03133"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/JetBrains-Research/codegen-metrics","reach":null}],"summary":{"ran_honours":2,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"cae2293698e4d173","entry":"ast_match","repo":"JetBrains-Research/codegen-metrics","repo_kind":"official","path":"metrics_evaluation/metrics/codebleu/codebleu.py","file_url":"https://github.com/JetBrains-Research/codegen-metrics/blob/HEAD/metrics_evaluation/metrics/codebleu/codebleu.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cae2293698e4d173"}},{"code_sha256_prefix":"cc2207e2b988a83d","entry":"dfg_match","repo":"JetBrains-Research/codegen-metrics","repo_kind":"official","path":"metrics_evaluation/metrics/codebleu/codebleu.py","file_url":"https://github.com/JetBrains-Research/codegen-metrics/blob/HEAD/metrics_evaluation/metrics/codebleu/codebleu.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cc2207e2b988a83d"}},{"code_sha256_prefix":"90b57eda04fc0532","entry":"tokenize_builtin","repo":"JetBrains-Research/codegen-metrics","repo_kind":"official","path":"metrics_evaluation/metrics/codebleu/codebleu.py","file_url":"https://github.com/JetBrains-Research/codegen-metrics/blob/HEAD/metrics_evaluation/metrics/codebleu/codebleu.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"90b57eda04fc0532"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}