{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sentencepiece-a-simple-and-language-1","title":"SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing","arxiv_id":"1808.06226","date":"2018-08-19","proceeding":"EMNLP 2018 11","authors":["Taku Kudo","John Richardson"],"abstract":"This paper describes SentencePiece, a language-independent subword tokenizer\nand detokenizer designed for Neural-based text processing, including Neural\nMachine Translation. It provides open-source C++ and Python implementations for\nsubword units. While existing subword segmentation tools assume that the input\nis pre-tokenized into word sequences, SentencePiece can train subword models\ndirectly from raw sentences, which allows us to make a purely end-to-end and\nlanguage independent system. We perform a validation experiment of NMT on\nEnglish-Japanese machine translation, and find that it is possible to achieve\ncomparable accuracy to direct subword training from raw sentences. We also\ncompare the performance of subword training and segmentation with various\nconfigurations. SentencePiece is available under the Apache 2 license at\nhttps://github.com/google/sentencepiece.","url_abs":"http://arxiv.org/abs/1808.06226v1","url_pdf":"http://arxiv.org/pdf/1808.06226v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sentencepiece-a-simple-and-language-1","repo_url":"https://github.com/google/sentencepiece","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"sentencepiece-a-simple-and-language-1","repo_url":"https://github.com/conceptofmind/LaMDA-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"sentencepiece-a-simple-and-language-1","repo_url":"https://github.com/conceptofmind/lamda-rlhf-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"nmt","task_name":"NMT"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"translation","task_name":"Translation"}],"methods":[{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"sentencepiece","method_name":"SentencePiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1808.06226","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}