{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/evaluating-style-transfer-for-text","title":"Evaluating Style Transfer for Text","arxiv_id":"1904.02295","date":"2019-04-04","proceeding":"NAACL 2019 6","authors":["Remi Mir","Bjarke Felbo","Nick Obradovich","Iyad Rahwan"],"abstract":"Research in the area of style transfer for text is currently bottlenecked by\na lack of standard evaluation practices. This paper aims to alleviate this\nissue by experimentally identifying best practices with a Yelp sentiment\ndataset. We specify three aspects of interest (style transfer intensity,\ncontent preservation, and naturalness) and show how to obtain more reliable\nmeasures of them from human evaluation than in previous work. We propose a set\nof metrics for automated evaluation and demonstrate that they are more strongly\ncorrelated and in agreement with human judgment: direction-corrected Earth\nMover's Distance, Word Mover's Distance on style-masked texts, and adversarial\nclassification for the respective aspects. We also show that the three examined\nmodels exhibit tradeoffs between aspects of interest, demonstrating the\nimportance of evaluating style transfer models at specific points of their\ntradeoff plots. We release software with our evaluation metrics to facilitate\nresearch.","url_abs":"http://arxiv.org/abs/1904.02295v1","url_pdf":"http://arxiv.org/pdf/1904.02295v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"evaluating-style-transfer-for-text","repo_url":"https://github.com/passeul/style-transfer-model-evaluation","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"style-transfer","task_name":"Style Transfer"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1904.02295","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}