{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/embarrassingly-easy-document-level-mt-metrics","title":"Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric Into a Document-Level Metric","arxiv_id":"2209.13654","date":"2022-09-27","proceeding":null,"authors":["Giorgos Vernikos","Brian Thompson","Prashant Mathur","Marcello Federico"],"abstract":"We hypothesize that existing sentence-level machine translation (MT) metrics become less effective when the human reference contains ambiguities. To verify this hypothesis, we present a very simple method for extending pretrained metrics to incorporate context at the document level. We apply our method to three popular metrics, BERTScore, Prism, and COMET, and to the reference free metric COMET-QE. We evaluate the extended metrics on the WMT 2021 metrics shared task using the provided MQM annotations. Our results show that the extended metrics outperform their sentence-level counterparts in about 85% of the tested conditions, when excluding results on low-quality human references. Additionally, we show that our document-level extension of COMET-QE dramatically improves its accuracy on discourse phenomena tasks, outperforming a dedicated baseline by up to 6.1%. Our experimental results support our initial hypothesis and show that a simple extension of the metrics permits them to take advantage of context to resolve ambiguities in the reference.","url_abs":"https://arxiv.org/abs/2209.13654v1","url_pdf":"https://arxiv.org/pdf/2209.13654v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"embarrassingly-easy-document-level-mt-metrics","repo_url":"https://github.com/amazon-science/doc-mt-metrics","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"sentence","task_name":"Sentence"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2209.13654","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2209.13654"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amazon-science/doc-mt-metrics","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"25bb285ada0f5ab2","entry":"add_context","repo":"amazon-science/doc-mt-metrics","repo_kind":"official","path":"bert_score/add_context.py","file_url":"https://github.com/amazon-science/doc-mt-metrics/blob/HEAD/bert_score/add_context.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"25bb285ada0f5ab2"}},{"code_sha256_prefix":"40f6add30e9986bf","entry":"get_data","repo":"amazon-science/doc-mt-metrics","repo_kind":"official","path":"bert_score/get_rescale_baseline/get_rescale_baseline.py","file_url":"https://github.com/amazon-science/doc-mt-metrics/blob/HEAD/bert_score/get_rescale_baseline/get_rescale_baseline.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"40f6add30e9986bf"}},{"code_sha256_prefix":"e655efd76f64a7aa","entry":"get_model","repo":"amazon-science/doc-mt-metrics","repo_kind":"official","path":"bert_score/bert_score/utils.py","file_url":"https://github.com/amazon-science/doc-mt-metrics/blob/HEAD/bert_score/bert_score/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e655efd76f64a7aa"}},{"code_sha256_prefix":"8d08178958f5f20d","entry":"get_tokenizer","repo":"amazon-science/doc-mt-metrics","repo_kind":"official","path":"bert_score/bert_score/utils.py","file_url":"https://github.com/amazon-science/doc-mt-metrics/blob/HEAD/bert_score/bert_score/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8d08178958f5f20d"}},{"code_sha256_prefix":"efd9b25cfcd27f97","entry":"score","repo":"amazon-science/doc-mt-metrics","repo_kind":"official","path":"bert_score/bert_score/score.py","file_url":"https://github.com/amazon-science/doc-mt-metrics/blob/HEAD/bert_score/bert_score/score.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"efd9b25cfcd27f97"}},{"code_sha256_prefix":"7aa792f0b1258927","entry":"sent_encode","repo":"amazon-science/doc-mt-metrics","repo_kind":"official","path":"bert_score/bert_score/utils.py","file_url":"https://github.com/amazon-science/doc-mt-metrics/blob/HEAD/bert_score/bert_score/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7aa792f0b1258927"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}