{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bleu-is-not-suitable-for-the-evaluation-of","title":"BLEU is Not Suitable for the Evaluation of Text Simplification","arxiv_id":"1810.05995","date":"2018-10-14","proceeding":"EMNLP 2018 10","authors":["Elior Sulem","Omri Abend","Ari Rappoport"],"abstract":"BLEU is widely considered to be an informative metric for text-to-text\ngeneration, including Text Simplification (TS). TS includes both lexical and\nstructural aspects. In this paper we show that BLEU is not suitable for the\nevaluation of sentence splitting, the major structural simplification\noperation. We manually compiled a sentence splitting gold standard corpus\ncontaining multiple structural paraphrases, and performed a correlation\nanalysis with human judgments. We find low or no correlation between BLEU and\nthe grammaticality and meaning preservation parameters where sentence splitting\nis involved. Moreover, BLEU often negatively correlates with simplicity,\nessentially penalizing simpler sentences.","url_abs":"http://arxiv.org/abs/1810.05995v1","url_pdf":"http://arxiv.org/pdf/1810.05995v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bleu-is-not-suitable-for-the-evaluation-of","repo_url":"https://github.com/eliorsulem/HSplit-corpus","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"text-simplification","task_name":"Text Simplification"}],"methods":[{"method_slug":"ts","method_name":"TS"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1810.05995","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}