{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-call-for-clarity-in-reporting-bleu-scores","title":"A Call for Clarity in Reporting BLEU Scores","arxiv_id":"1804.08771","date":"2018-04-23","proceeding":"WS 2018 10","authors":["Matt Post"],"abstract":"The field of machine translation faces an under-recognized problem because of\ninconsistency in the reporting of scores from its dominant metric. Although\npeople refer to \"the\" BLEU score, BLEU is in fact a parameterized metric whose\nvalues can vary wildly with changes to these parameters. These parameters are\noften not reported or are hard to find, and consequently, BLEU scores between\npapers cannot be directly compared. I quantify this variation, finding\ndifferences as high as 1.8 between commonly used configurations. The main\nculprit is different tokenization and normalization schemes applied to the\nreference. Pointing to the success of the parsing community, I suggest machine\ntranslation researchers settle upon the BLEU scheme used by the annual\nConference on Machine Translation (WMT), which does not allow for user-supplied\nreference processing, and provide a new tool, SacreBLEU, to facilitate this.","url_abs":"http://arxiv.org/abs/1804.08771v2","url_pdf":"http://arxiv.org/pdf/1804.08771v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-call-for-clarity-in-reporting-bleu-scores","repo_url":"https://github.com/jchenghu/sacreeos","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"a-call-for-clarity-in-reporting-bleu-scores","repo_url":"https://github.com/mjpost/sacreBLEU","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"translation","task_name":"Translation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1804.08771","atlas_url":"https://app.syntology.ai/?focus=1804.08771","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1804.08771"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mjpost/sacreBLEU","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jchenghu/sacreeos","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4,"unverified":7},"by_repo_kind":{"listed":{"samples":11,"ran":4,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e8aa5f602ca91360","entry":"estimate_ci","repo":"mjpost/sacreBLEU","repo_kind":"listed","path":"sacrebleu/significance.py","file_url":"https://github.com/mjpost/sacreBLEU/blob/HEAD/sacrebleu/significance.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e8aa5f602ca91360"}},{"code_sha256_prefix":"a37d553ae5141065","entry":"my_log","repo":"mjpost/sacreBLEU","repo_kind":"listed","path":"sacrebleu/utils.py","file_url":"https://github.com/mjpost/sacreBLEU/blob/HEAD/sacrebleu/utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a37d553ae5141065"}},{"code_sha256_prefix":"09bfbf9dc612b494","entry":"smart_open","repo":"mjpost/sacreBLEU","repo_kind":"listed","path":"sacrebleu/utils.py","file_url":"https://github.com/mjpost/sacreBLEU/blob/HEAD/sacrebleu/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"09bfbf9dc612b494"}},{"code_sha256_prefix":"67d5f0d11404b388","entry":"sum_of_lists","repo":"mjpost/sacreBLEU","repo_kind":"listed","path":"sacrebleu/utils.py","file_url":"https://github.com/mjpost/sacreBLEU/blob/HEAD/sacrebleu/utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"67d5f0d11404b388"}},{"code_sha256_prefix":"9ac2d70f97a531f2","entry":"compute_doc_freq","repo":"jchenghu/sacreeos","repo_kind":"listed","path":"sacreeos/metric/py_core/py_cider_scorer.py","file_url":"https://github.com/jchenghu/sacreeos/blob/HEAD/sacreeos/metric/py_core/py_cider_scorer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9ac2d70f97a531f2"}},{"code_sha256_prefix":"b1a46356ad1589f8","entry":"compute_penalty_by_length","repo":"jchenghu/sacreeos","repo_kind":"listed","path":"sacreeos/metric/py_core/py_cider_scorer.py","file_url":"https://github.com/jchenghu/sacreeos/blob/HEAD/sacreeos/metric/py_core/py_cider_scorer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b1a46356ad1589f8"}},{"code_sha256_prefix":"993ef917c2a9a9de","entry":"compute_penalty_by_repetition","repo":"jchenghu/sacreeos","repo_kind":"listed","path":"sacreeos/metric/py_core/py_cider_scorer.py","file_url":"https://github.com/jchenghu/sacreeos/blob/HEAD/sacreeos/metric/py_core/py_cider_scorer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"993ef917c2a9a9de"}},{"code_sha256_prefix":"b56d6d3f3ce198f0","entry":"cook_refss","repo":"jchenghu/sacreeos","repo_kind":"listed","path":"sacreeos/metric/py_core/ngram_utils.py","file_url":"https://github.com/jchenghu/sacreeos/blob/HEAD/sacreeos/metric/py_core/ngram_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b56d6d3f3ce198f0"}},{"code_sha256_prefix":"da4769829a3a7e22","entry":"cook_tests","repo":"jchenghu/sacreeos","repo_kind":"listed","path":"sacreeos/metric/py_core/ngram_utils.py","file_url":"https://github.com/jchenghu/sacreeos/blob/HEAD/sacreeos/metric/py_core/ngram_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"da4769829a3a7e22"}},{"code_sha256_prefix":"d744bcac482b9dbe","entry":"invoke_bleu_computer","repo":"jchenghu/sacreeos","repo_kind":"listed","path":"sacreeos/metric/c_core/invoker.py","file_url":"https://github.com/jchenghu/sacreeos/blob/HEAD/sacreeos/metric/c_core/invoker.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d744bcac482b9dbe"}},{"code_sha256_prefix":"ca007f4b8a45fbdc","entry":"precook","repo":"jchenghu/sacreeos","repo_kind":"listed","path":"sacreeos/metric/py_core/ngram_utils.py","file_url":"https://github.com/jchenghu/sacreeos/blob/HEAD/sacreeos/metric/py_core/ngram_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ca007f4b8a45fbdc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}