{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/aces-translation-accuracy-challenge-sets-for","title":"ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metrics","arxiv_id":"2210.15615","date":"2022-10-27","proceeding":null,"authors":["Chantal Amrhein","Nikita Moghe","Liane Guillou"],"abstract":"As machine translation (MT) metrics improve their correlation with human judgement every year, it is crucial to understand the limitations of such metrics at the segment level. Specifically, it is important to investigate metric behaviour when facing accuracy errors in MT because these can have dangerous consequences in certain contexts (e.g., legal, medical). We curate ACES, a translation accuracy challenge set, consisting of 68 phenomena ranging from simple perturbations at the word/character level to more complex errors based on discourse and real-world knowledge. We use ACES to evaluate a wide range of MT metrics including the submissions to the WMT 2022 metrics shared task and perform several analyses leading to general recommendations for metric developers. We recommend: a) combining metrics with different strengths, b) developing metrics that give more weight to the source and less to surface-level overlap with the reference and c) explicitly modelling additional language-specific information beyond what is available via multilingual embeddings.","url_abs":"https://arxiv.org/abs/2210.15615v2","url_pdf":"https://arxiv.org/pdf/2210.15615v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"aces-translation-accuracy-challenge-sets-for","repo_url":"https://github.com/edinburghnlp/aces","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"translation","task_name":"Translation"},{"task_slug":"world-knowledge","task_name":"World Knowledge"}],"methods":[],"datasets_introduced":[{"slug":"aces","name":"ACES","full_name":"A Translation Accuracy Challenge Set"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/machine-translation-on-aces","task":"Machine Translation","dataset":"ACES","model":"HWTSC-Teacher-Sim","rank_in_archive_order":1,"of":21,"metrics":{"Score":"19.97"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-aces","task":"Machine Translation","dataset":"ACES","model":"MS-COMET-22","rank_in_archive_order":2,"of":21,"metrics":{"Score":"19.89"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-aces","task":"Machine Translation","dataset":"ACES","model":"MS-COMET-QE-22","rank_in_archive_order":3,"of":21,"metrics":{"Score":"19.76"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-aces","task":"Machine Translation","dataset":"ACES","model":"KG-BERTScore","rank_in_archive_order":4,"of":21,"metrics":{"Score":"17.28"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-aces","task":"Machine Translation","dataset":"ACES","model":"metricx_xl_DA_2019","rank_in_archive_order":5,"of":21,"metrics":{"Score":"17.17"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-aces","task":"Machine Translation","dataset":"ACES","model":"COMET-QE","rank_in_archive_order":6,"of":21,"metrics":{"Score":"16.8"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-aces","task":"Machine Translation","dataset":"ACES","model":"COMET-22","rank_in_archive_order":7,"of":21,"metrics":{"Score":"16.31"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-aces","task":"Machine Translation","dataset":"ACES","model":"UniTE-src","rank_in_archive_order":8,"of":21,"metrics":{"Score":"15.68"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-aces","task":"Machine Translation","dataset":"ACES","model":"UniTE-ref","rank_in_archive_order":9,"of":21,"metrics":{"Score":"15.38"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-aces","task":"Machine Translation","dataset":"ACES","model":"metricx_xxl_DA_2019","rank_in_archive_order":10,"of":21,"metrics":{"Score":"15.24"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-aces","task":"Machine Translation","dataset":"ACES","model":"UniTE","rank_in_archive_order":11,"of":21,"metrics":{"Score":"14.76"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-aces","task":"Machine Translation","dataset":"ACES","model":"Cross-QE","rank_in_archive_order":12,"of":21,"metrics":{"Score":"14.07"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-aces","task":"Machine Translation","dataset":"ACES","model":"chrF","rank_in_archive_order":13,"of":21,"metrics":{"Score":"13.57"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-aces","task":"Machine Translation","dataset":"ACES","model":"metricx_xl_MQM_2020","rank_in_archive_order":14,"of":21,"metrics":{"Score":"13.08"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-aces","task":"Machine Translation","dataset":"ACES","model":"COMET-20","rank_in_archive_order":15,"of":21,"metrics":{"Score":"12.06"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-aces","task":"Machine Translation","dataset":"ACES","model":"BLEURT-20","rank_in_archive_order":16,"of":21,"metrics":{"Score":"11.9"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-aces","task":"Machine Translation","dataset":"ACES","model":"YiSi-1","rank_in_archive_order":17,"of":21,"metrics":{"Score":"11.38"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-aces","task":"Machine Translation","dataset":"ACES","model":"BERTScore","rank_in_archive_order":18,"of":21,"metrics":{"Score":"10.47"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-aces","task":"Machine Translation","dataset":"ACES","model":"BLEU","rank_in_archive_order":19,"of":21,"metrics":{"Score":"-3.13"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-aces","task":"Machine Translation","dataset":"ACES","model":"f101spBLEU","rank_in_archive_order":20,"of":21,"metrics":{"Score":"-0.33"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-aces","task":"Machine Translation","dataset":"ACES","model":"f200spBLEU","rank_in_archive_order":21,"of":21,"metrics":{"Score":"-0.18"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.15615","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.15615"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/edinburghnlp/aces","reach":null}],"summary":{"ran_honours":1,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"bbcfcf7cdfa562ac","entry":"comp_aces_score","repo":"edinburghnlp/aces","repo_kind":"official","path":"aces/cli/evaluate.py","file_url":"https://github.com/edinburghnlp/aces/blob/HEAD/aces/cli/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bbcfcf7cdfa562ac"}},{"code_sha256_prefix":"413ea7f78d43e011","entry":"comp_corr","repo":"edinburghnlp/aces","repo_kind":"official","path":"aces/cli/evaluate.py","file_url":"https://github.com/edinburghnlp/aces/blob/HEAD/aces/cli/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"413ea7f78d43e011"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}