{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/navigating-the-metrics-maze-reconciling-score","title":"Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies","arxiv_id":"2401.06760","date":"2024-01-12","proceeding":null,"authors":["Tom Kocmi","Vilém Zouhar","Christian Federmann","Matt Post"],"abstract":"Ten years ago a single metric, BLEU, governed progress in machine translation research. For better or worse, there is no such consensus today, and consequently it is difficult for researchers to develop and retain the kinds of heuristic intuitions about metric deltas that drove earlier research and deployment decisions. This paper investigates the \"dynamic range\" of a number of modern metrics in an effort to provide a collective understanding of the meaning of differences in scores both within and among metrics; in other words, we ask what point difference X in metric Y is required between two systems for humans to notice? We conduct our evaluation on a new large dataset, ToShip23, using it to discover deltas at which metrics achieve system-level differences that are meaningful to humans, which we measure by pairwise system accuracy. We additionally show that this method of establishing delta-accuracy is more stable than the standard use of statistical p-values in regards to testset size. Where data size permits, we also explore the effect of metric deltas and accuracy across finer-grained features such as translation direction, domain, and system closeness.","url_abs":"https://arxiv.org/abs/2401.06760v2","url_pdf":"https://arxiv.org/pdf/2401.06760v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"navigating-the-metrics-maze-reconciling-score","repo_url":"https://github.com/kocmitom/mt-thresholds","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"navigating-the-metrics-maze-reconciling-score","repo_url":"https://github.com/Unbabel/COMET","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"translation","task_name":"Translation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.06760","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.06760"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Unbabel/COMET","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kocmitom/mt-thresholds","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/kocmitom/MT-Thresholds","reach":{"status":"ok"}}],"summary":{"ran_honours":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"126fd4729a703e2a","entry":"accuracy","repo":"kocmitom/mt-thresholds","repo_kind":"official","path":"python-tool/mt_thresholds.py","file_url":"https://github.com/kocmitom/mt-thresholds/blob/HEAD/python-tool/mt_thresholds.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"126fd4729a703e2a"}},{"code_sha256_prefix":"7d6a35ed50115395","entry":"delta","repo":"kocmitom/mt-thresholds","repo_kind":"official","path":"python-tool/mt_thresholds.py","file_url":"https://github.com/kocmitom/mt-thresholds/blob/HEAD/python-tool/mt_thresholds.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7d6a35ed50115395"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}