{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/an-empirical-study-of-tokenization-strategies","title":"An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks","arxiv_id":"2010.02534","date":"2020-10-06","proceeding":"Asian Chapter of the Association for Computational Linguistics 2020","authors":["Kyubyong Park","Joohong Lee","Seongbo Jang","Dawoon Jung"],"abstract":"Typically, tokenization is the very first step in most text processing works. As a token serves as an atomic unit that embeds the contextual information of text, how to define a token plays a decisive role in the performance of a model.Even though Byte Pair Encoding (BPE) has been considered the de facto standard tokenization method due to its simplicity and universality, it still remains unclear whether BPE works best across all languages and tasks. In this paper, we test several tokenization strategies in order to answer our primary research question, that is, \"What is the best tokenization strategy for Korean NLP tasks?\" Experimental results demonstrate that a hybrid approach of morphological segmentation followed by BPE works best in Korean to/from English machine translation and natural language understanding tasks such as KorNLI, KorSTS, NSMC, and PAWS-X. As an exception, for KorQuAD, the Korean extension of SQuAD, BPE segmentation turns out to be the most effective.","url_abs":"https://arxiv.org/abs/2010.02534v1","url_pdf":"https://arxiv.org/pdf/2010.02534v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"an-empirical-study-of-tokenization-strategies","repo_url":"https://github.com/kakaobrain/kortok","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"natural-language-understanding","task_name":"Natural Language Understanding"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"translation","task_name":"Translation"}],"methods":[{"method_slug":"bpe","method_name":"BPE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2010.02534","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2010.02534"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kakaobrain/kortok","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":2},"by_repo_kind":{"official":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8b586b3fbd2c898f","entry":"get_logger","repo":"kakaobrain/kortok","repo_kind":"official","path":"tasks/logger.py","file_url":"https://github.com/kakaobrain/kortok/blob/HEAD/tasks/logger.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8b586b3fbd2c898f"}},{"code_sha256_prefix":"f8dfe07c8e3fc107","entry":"load_pretrained_bert","repo":"kakaobrain/kortok","repo_kind":"official","path":"tasks/bert_utils.py","file_url":"https://github.com/kakaobrain/kortok/blob/HEAD/tasks/bert_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f8dfe07c8e3fc107"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}