{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/which-encoding-is-the-best-for-text","title":"Which Encoding is the Best for Text Classification in Chinese, English, Japanese and Korean?","arxiv_id":"1708.02657","date":"2017-08-08","proceeding":null,"authors":["Xiang Zhang","Yann Lecun"],"abstract":"This article offers an empirical study on the different ways of encoding\nChinese, Japanese, Korean (CJK) and English languages for text classification.\nDifferent encoding levels are studied, including UTF-8 bytes, characters,\nwords, romanized characters and romanized words. For all encoding levels,\nwhenever applicable, we provide comparisons with linear models, fastText and\nconvolutional networks. For convolutional networks, we compare between encoding\nmechanisms using character glyph images, one-hot (or one-of-n) encoding, and\nembedding. In total there are 473 models, using 14 large-scale text\nclassification datasets in 4 languages including Chinese, English, Japanese and\nKorean. Some conclusions from these results include that byte-level one-hot\nencoding based on UTF-8 consistently produces competitive results for\nconvolutional networks, that word-level n-grams linear models are competitive\neven without perfect word segmentation, and that fastText provides the best\nresult using character-level n-gram encoding but can overfit when the features\nare overly rich.","url_abs":"http://arxiv.org/abs/1708.02657v2","url_pdf":"http://arxiv.org/pdf/1708.02657v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"which-encoding-is-the-best-for-text","repo_url":"https://github.com/Nov05/Genre-Fiction-Classification","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"which-encoding-is-the-best-for-text","repo_url":"https://github.com/dbiir/UER-py","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"which-encoding-is-the-best-for-text","repo_url":"https://github.com/zhangxiangxiao/glyph","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"classification","task_name":"General Classification"},{"task_slug":"text-classification","task_name":"Text Classification"}],"methods":[{"method_slug":"fasttext","method_name":"fastText"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1708.02657","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}