{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-devil-is-in-the-errors-leveraging-large","title":"The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation","arxiv_id":"2308.07286","date":"2023-08-14","proceeding":null,"authors":["Patrick Fernandes","Daniel Deutsch","Mara Finkelstein","Parker Riley","André F. T. Martins","Graham Neubig","Ankush Garg","Jonathan H. Clark","Markus Freitag","Orhan Firat"],"abstract":"Automatic evaluation of machine translation (MT) is a critical tool driving the rapid iterative development of MT systems. While considerable progress has been made on estimating a single scalar quality score, current metrics lack the informativeness of more detailed schemes that annotate individual errors, such as Multidimensional Quality Metrics (MQM). In this paper, we help fill this gap by proposing AutoMQM, a prompting technique which leverages the reasoning and in-context learning capabilities of large language models (LLMs) and asks them to identify and categorize errors in translations. We start by evaluating recent LLMs, such as PaLM and PaLM-2, through simple score prediction prompting, and we study the impact of labeled data through in-context learning and finetuning. We then evaluate AutoMQM with PaLM-2 models, and we find that it improves performance compared to just prompting for scores (with particularly large gains for larger models) while providing interpretability through error spans that align with human annotations.","url_abs":"https://arxiv.org/abs/2308.07286v1","url_pdf":"https://arxiv.org/pdf/2308.07286v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-devil-is-in-the-errors-leveraging-large","repo_url":"https://github.com/xuuhuang/lost_in_the_src","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"informativeness","task_name":"Informativeness"},{"task_slug":"machine-translation","task_name":"Machine Translation"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"palm","method_name":"PaLM"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.07286","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.07286"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xuuhuang/lost_in_the_src","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":12},"by_repo_kind":{"listed":{"samples":13,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"817d5f759487007d","entry":"preprocess","repo":"xuuhuang/lost_in_the_src","repo_kind":"listed","path":"automqm/finetune_llama.py","file_url":"https://github.com/xuuhuang/lost_in_the_src/blob/HEAD/automqm/finetune_llama.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"817d5f759487007d"}},{"code_sha256_prefix":"ceffa1ba9496bac6","entry":"check_icl_set","repo":"xuuhuang/lost_in_the_src","repo_kind":"listed","path":"automqm/rejection_sample.py","file_url":"https://github.com/xuuhuang/lost_in_the_src/blob/HEAD/automqm/rejection_sample.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ceffa1ba9496bac6"}},{"code_sha256_prefix":"ca6b7c2e99c13a8b","entry":"check_icl_set_uniform","repo":"xuuhuang/lost_in_the_src","repo_kind":"listed","path":"automqm/rejection_sample.py","file_url":"https://github.com/xuuhuang/lost_in_the_src/blob/HEAD/automqm/rejection_sample.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ca6b7c2e99c13a8b"}},{"code_sha256_prefix":"d521942f2e0519a5","entry":"format_examples","repo":"xuuhuang/lost_in_the_src","repo_kind":"listed","path":"automqm/automqm.py","file_url":"https://github.com/xuuhuang/lost_in_the_src/blob/HEAD/automqm/automqm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d521942f2e0519a5"}},{"code_sha256_prefix":"1790c7db877d5562","entry":"format_prefix","repo":"xuuhuang/lost_in_the_src","repo_kind":"listed","path":"automqm/llm_ppl.py","file_url":"https://github.com/xuuhuang/lost_in_the_src/blob/HEAD/automqm/llm_ppl.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1790c7db877d5562"}},{"code_sha256_prefix":"e9ffc8aced13263c","entry":"format_template","repo":"xuuhuang/lost_in_the_src","repo_kind":"listed","path":"automqm/automqm.py","file_url":"https://github.com/xuuhuang/lost_in_the_src/blob/HEAD/automqm/automqm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e9ffc8aced13263c"}},{"code_sha256_prefix":"5d2cdb987c833aa0","entry":"initialize","repo":"xuuhuang/lost_in_the_src","repo_kind":"listed","path":"automqm/llm_ppl.py","file_url":"https://github.com/xuuhuang/lost_in_the_src/blob/HEAD/automqm/llm_ppl.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5d2cdb987c833aa0"}},{"code_sha256_prefix":"4fa0d7896f0984c2","entry":"load_file","repo":"xuuhuang/lost_in_the_src","repo_kind":"listed","path":"automqm/llm_ppl.py","file_url":"https://github.com/xuuhuang/lost_in_the_src/blob/HEAD/automqm/llm_ppl.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4fa0d7896f0984c2"}},{"code_sha256_prefix":"78553e11ab537e05","entry":"make_supervised_data_module","repo":"xuuhuang/lost_in_the_src","repo_kind":"listed","path":"automqm/finetune_llama.py","file_url":"https://github.com/xuuhuang/lost_in_the_src/blob/HEAD/automqm/finetune_llama.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"78553e11ab537e05"}},{"code_sha256_prefix":"8d0012e897a9cfff","entry":"preprocess_results","repo":"xuuhuang/lost_in_the_src","repo_kind":"listed","path":"automqm/evaluate.py","file_url":"https://github.com/xuuhuang/lost_in_the_src/blob/HEAD/automqm/evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8d0012e897a9cfff"}},{"code_sha256_prefix":"859ef0e2fe47e47b","entry":"process_type","repo":"xuuhuang/lost_in_the_src","repo_kind":"listed","path":"automqm/evaluate.py","file_url":"https://github.com/xuuhuang/lost_in_the_src/blob/HEAD/automqm/evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"859ef0e2fe47e47b"}},{"code_sha256_prefix":"be66b9284a0d7bbd","entry":"score_label","repo":"xuuhuang/lost_in_the_src","repo_kind":"listed","path":"automqm/postprocess_inference.py","file_url":"https://github.com/xuuhuang/lost_in_the_src/blob/HEAD/automqm/postprocess_inference.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"be66b9284a0d7bbd"}},{"code_sha256_prefix":"29de5dbe6b96b9f4","entry":"span_label","repo":"xuuhuang/lost_in_the_src","repo_kind":"listed","path":"automqm/evaluate.py","file_url":"https://github.com/xuuhuang/lost_in_the_src/blob/HEAD/automqm/evaluate.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"29de5dbe6b96b9f4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}