{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/xcomet-transparent-machine-translation","title":"xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection","arxiv_id":"2310.10482","date":"2023-10-16","proceeding":null,"authors":["Nuno M. Guerreiro","Ricardo Rei","Daan van Stigt","Luisa Coheur","Pierre Colombo","André F. T. Martins"],"abstract":"Widely used learned metrics for machine translation evaluation, such as COMET and BLEURT, estimate the quality of a translation hypothesis by providing a single sentence-level score. As such, they offer little insight into translation errors (e.g., what are the errors and what is their severity). On the other hand, generative large language models (LLMs) are amplifying the adoption of more granular strategies to evaluation, attempting to detail and categorize translation errors. In this work, we introduce xCOMET, an open-source learned metric designed to bridge the gap between these approaches. xCOMET integrates both sentence-level evaluation and error span detection capabilities, exhibiting state-of-the-art performance across all types of evaluation (sentence-level, system-level, and error span detection). Moreover, it does so while highlighting and categorizing error spans, thus enriching the quality assessment. We also provide a robustness analysis with stress tests, and show that xCOMET is largely capable of identifying localized critical errors and hallucinations.","url_abs":"https://arxiv.org/abs/2310.10482v1","url_pdf":"https://arxiv.org/pdf/2310.10482v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"xcomet-transparent-machine-translation","repo_url":"https://github.com/PinzhenChen/sacreCOMET","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"xcomet-transparent-machine-translation","repo_url":"https://github.com/Unbabel/COMET","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"xcomet-transparent-machine-translation","repo_url":"https://github.com/coldmist-lu/mqm_ape","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"translation","task_name":"Translation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.10482","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.10482"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PinzhenChen/sacreCOMET","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Unbabel/COMET","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/coldmist-lu/mqm_ape","reach":{"status":"ok"}}],"summary":{"ran_violates":1,"ran_honours":1,"ran_draft_wrong":1},"by_repo_kind":{"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b32b528858bf1fbb","entry":"MAE","repo":"PinzhenChen/sacreCOMET","repo_kind":"listed","path":"experiments/precision/compute_metric_corr.py","file_url":"https://github.com/PinzhenChen/sacreCOMET/blob/HEAD/experiments/precision/compute_metric_corr.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b32b528858bf1fbb"}},{"code_sha256_prefix":"f1410b27c8c0b6f3","entry":"read_tsv","repo":"PinzhenChen/sacreCOMET","repo_kind":"listed","path":"experiments/precision/compute_metric_corr.py","file_url":"https://github.com/PinzhenChen/sacreCOMET/blob/HEAD/experiments/precision/compute_metric_corr.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1410b27c8c0b6f3"}},{"code_sha256_prefix":"f62421f9e30043cb","entry":"read_tsv_for_pairwise_accuracy","repo":"PinzhenChen/sacreCOMET","repo_kind":"listed","path":"experiments/precision/compute_metric_corr.py","file_url":"https://github.com/PinzhenChen/sacreCOMET/blob/HEAD/experiments/precision/compute_metric_corr.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f62421f9e30043cb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}