{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/otemutem-over-and-under-translation","title":"Otem&Utem: Over- and Under-Translation Evaluation Metric for NMT","arxiv_id":"1807.08945","date":"2018-07-24","proceeding":null,"authors":["Jing Yang","Biao Zhang","Yue Qin","Xiangwen Zhang","Qian Lin","Jinsong Su"],"abstract":"Although neural machine translation(NMT) yields promising translation\nperformance, it unfortunately suffers from over- and under-translation is- sues\n[Tu et al., 2016], of which studies have become research hotspots in NMT. At\npresent, these studies mainly apply the dominant automatic evaluation metrics,\nsuch as BLEU, to evaluate the overall translation quality with respect to both\nadequacy and uency. However, they are unable to accurately measure the ability\nof NMT systems in dealing with the above-mentioned issues. In this paper, we\npropose two quantitative metrics, the Otem and Utem, to automatically evaluate\nthe system perfor- mance in terms of over- and under-translation respectively.\nBoth metrics are based on the proportion of mismatched n-grams between gold\nref- erence and system translation. We evaluate both metrics by comparing their\nscores with human evaluations, where the values of Pearson Cor- relation\nCoefficient reveal their strong correlation. Moreover, in-depth analyses on\nvarious translation systems indicate some inconsistency be- tween BLEU and our\nproposed metrics, highlighting the necessity and significance of our metrics.","url_abs":"http://arxiv.org/abs/1807.08945v1","url_pdf":"http://arxiv.org/pdf/1807.08945v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"otemutem-over-and-under-translation","repo_url":"https://github.com/DeepLearnXMU/Otem-Utem","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"nmt","task_name":"NMT"},{"task_slug":"translation","task_name":"Translation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1807.08945","atlas_url":"https://app.syntology.ai/?focus=1807.08945","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}