{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/191013267","title":"BPE-Dropout: Simple and Effective Subword Regularization","arxiv_id":"1910.13267","date":"2019-10-29","proceeding":"ACL 2020 6","authors":["Ivan Provilkov","Dmitrii Emelianenko","Elena Voita"],"abstract":"Subword segmentation is widely used to address the open vocabulary problem in machine translation. The dominant approach to subword segmentation is Byte Pair Encoding (BPE), which keeps the most frequent words intact while splitting the rare ones into multiple tokens. While multiple segmentations are possible even with the same vocabulary, BPE splits words into unique sequences; this may prevent a model from better learning the compositionality of words and being robust to segmentation errors. So far, the only way to overcome this BPE imperfection, its deterministic nature, was to create another subword segmentation algorithm (Kudo, 2018). In contrast, we show that BPE itself incorporates the ability to produce multiple segmentations of the same word. We introduce BPE-dropout - simple and effective subword regularization method based on and compatible with conventional BPE. It stochastically corrupts the segmentation procedure of BPE, which leads to producing multiple segmentations within the same fixed BPE framework. Using BPE-dropout during training and the standard BPE during inference improves translation quality up to 3 BLEU compared to BPE and up to 0.9 BLEU compared to the previous subword regularization.","url_abs":"https://arxiv.org/abs/1910.13267v2","url_pdf":"https://arxiv.org/pdf/1910.13267v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"191013267","repo_url":"https://github.com/VProv/BPE-Dropout","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":null},{"paper_slug":"191013267","repo_url":"https://github.com/PatxiofromAlphensign/subword-nmt-mods","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"191013267","repo_url":"https://github.com/VKCOM/YouTokenToMe","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"191013267","repo_url":"https://github.com/google/sentencepiece","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"191013267","repo_url":"https://github.com/kh-mo/QA_wikisql","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"191013267","repo_url":"https://github.com/mozilla/subword-nmt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"191013267","repo_url":"https://github.com/rsennrich/subword-nmt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"translation","task_name":"Translation"}],"methods":[{"method_slug":"bpe","method_name":"BPE"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/machine-translation-on-iwslt2015-english-1","task":"Machine Translation","dataset":"IWSLT2015 English-Vietnamese","model":"Transformer+BPE-dropout","rank_in_archive_order":3,"of":11,"metrics":{"BLEU":"33.27"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-iwslt2017-arabic","task":"Machine Translation","dataset":"IWSLT2017 Arabic-English","model":"Transformer base + BPE-Dropout","rank_in_archive_order":1,"of":2,"metrics":{"Cased sacreBLEU":"33.0"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-iwslt2017-english-1","task":"Machine Translation","dataset":"IWSLT2017 English-Arabic","model":"Transformer base + BPE-Dropout","rank_in_archive_order":1,"of":2,"metrics":{"Cased sacreBLEU":"15.2"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-iwslt2017-english","task":"Machine Translation","dataset":"IWSLT2017 English-French","model":"Transformer base + BPE-Dropout","rank_in_archive_order":1,"of":2,"metrics":{"Cased sacreBLEU":"39.83"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-iwslt2017-french","task":"Machine Translation","dataset":"IWSLT2017 French-English","model":"Transformer base + BPE-Dropout","rank_in_archive_order":1,"of":2,"metrics":{"Cased sacreBLEU":"38.6"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1910.13267","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1910.13267"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google/sentencepiece","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/VProv/BPE-Dropout","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PatxiofromAlphensign/subword-nmt-mods","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/VKCOM/YouTokenToMe","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mozilla/subword-nmt","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rsennrich/subword-nmt","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kh-mo/QA_wikisql","reach":null}],"summary":{"ran_violates":1,"ran_draft_wrong":7,"ran_fixture":1,"ran":1,"unverified":14},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1},"listed":{"samples":21,"ran":7,"repositories":4}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a0deb98db94ae939","entry":"check_vocab_and_split","repo":"rsennrich/subword-nmt","repo_kind":"listed","path":"subword_nmt/apply_bpe.py","file_url":"https://github.com/rsennrich/subword-nmt/blob/HEAD/subword_nmt/apply_bpe.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a0deb98db94ae939"}},{"code_sha256_prefix":"1dc5d59fd80a7419","entry":"create_parser","repo":"rsennrich/subword-nmt","repo_kind":"listed","path":"subword_nmt/learn_bpe.py","file_url":"https://github.com/rsennrich/subword-nmt/blob/HEAD/subword_nmt/learn_bpe.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1dc5d59fd80a7419"}},{"code_sha256_prefix":"f9d4ba98fc7a6214","entry":"create_parser","repo":"rsennrich/subword-nmt","repo_kind":"listed","path":"subword_nmt/apply_bpe.py","file_url":"https://github.com/rsennrich/subword-nmt/blob/HEAD/subword_nmt/apply_bpe.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9d4ba98fc7a6214"}},{"code_sha256_prefix":"932ffa4dce8fc503","entry":"encode","repo":"rsennrich/subword-nmt","repo_kind":"listed","path":"subword_nmt/apply_bpe.py","file_url":"https://github.com/rsennrich/subword-nmt/blob/HEAD/subword_nmt/apply_bpe.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"932ffa4dce8fc503"}},{"code_sha256_prefix":"fe96394f96a9c638","entry":"get_pair_statistics","repo":"rsennrich/subword-nmt","repo_kind":"listed","path":"subword_nmt/learn_bpe.py","file_url":"https://github.com/rsennrich/subword-nmt/blob/HEAD/subword_nmt/learn_bpe.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fe96394f96a9c638"}},{"code_sha256_prefix":"b83d785ca99bfc9f","entry":"get_vocabulary","repo":"rsennrich/subword-nmt","repo_kind":"listed","path":"subword_nmt/learn_bpe.py","file_url":"https://github.com/rsennrich/subword-nmt/blob/HEAD/subword_nmt/learn_bpe.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b83d785ca99bfc9f"}},{"code_sha256_prefix":"bc7574327ea1f634","entry":"get_vocabulary","repo":"mozilla/subword-nmt","repo_kind":"listed","path":"subword_nmt/learn_bpe.py","file_url":"https://github.com/mozilla/subword-nmt/blob/HEAD/subword_nmt/learn_bpe.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc7574327ea1f634"}},{"code_sha256_prefix":"990d82424d309a3b","entry":"load_merge_table","repo":"VProv/BPE-Dropout","repo_kind":"official","path":"bpe.py","file_url":"https://github.com/VProv/BPE-Dropout/blob/HEAD/bpe.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"990d82424d309a3b"}},{"code_sha256_prefix":"03e9a6a74a57cc23","entry":"load_subword_nmt_table","repo":"VProv/BPE-Dropout","repo_kind":"official","path":"bpe.py","file_url":"https://github.com/VProv/BPE-Dropout/blob/HEAD/bpe.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"03e9a6a74a57cc23"}},{"code_sha256_prefix":"d7a0c20cde2f68a7","entry":"tokenize_word","repo":"VProv/BPE-Dropout","repo_kind":"official","path":"bpe.py","file_url":"https://github.com/VProv/BPE-Dropout/blob/HEAD/bpe.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d7a0c20cde2f68a7"}},{"code_sha256_prefix":"68576350e37086cd","entry":"create_parser","repo":"PatxiofromAlphensign/subword-nmt-mods","repo_kind":"listed","path":"subword_nmt/apply_bpe.py","file_url":"https://github.com/PatxiofromAlphensign/subword-nmt-mods/blob/HEAD/subword_nmt/apply_bpe.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"68576350e37086cd"}},{"code_sha256_prefix":"640ae72885772256","entry":"create_parser","repo":"PatxiofromAlphensign/subword-nmt-mods","repo_kind":"listed","path":"subword_nmt/get_vocab.py","file_url":"https://github.com/PatxiofromAlphensign/subword-nmt-mods/blob/HEAD/subword_nmt/get_vocab.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"640ae72885772256"}},{"code_sha256_prefix":"c330a23c53f2f1b3","entry":"create_parser","repo":"PatxiofromAlphensign/subword-nmt-mods","repo_kind":"listed","path":"subword_nmt/learn_bpe.py","file_url":"https://github.com/PatxiofromAlphensign/subword-nmt-mods/blob/HEAD/subword_nmt/learn_bpe.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c330a23c53f2f1b3"}},{"code_sha256_prefix":"e4ba5660a014dc1f","entry":"create_parser","repo":"PatxiofromAlphensign/subword-nmt-mods","repo_kind":"listed","path":"subword_nmt/learn_joint_bpe_and_vocab.py","file_url":"https://github.com/PatxiofromAlphensign/subword-nmt-mods/blob/HEAD/subword_nmt/learn_joint_bpe_and_vocab.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e4ba5660a014dc1f"}},{"code_sha256_prefix":"3a29f95d3b790d29","entry":"create_parser","repo":"PatxiofromAlphensign/subword-nmt-mods","repo_kind":"listed","path":"subword_nmt/segment_char_ngrams.py","file_url":"https://github.com/PatxiofromAlphensign/subword-nmt-mods/blob/HEAD/subword_nmt/segment_char_ngrams.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3a29f95d3b790d29"}},{"code_sha256_prefix":"3a18c84fbdddf7b4","entry":"create_parser","repo":"mozilla/subword-nmt","repo_kind":"listed","path":"subword_nmt/apply_bpe.py","file_url":"https://github.com/mozilla/subword-nmt/blob/HEAD/subword_nmt/apply_bpe.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3a18c84fbdddf7b4"}},{"code_sha256_prefix":"049cda35c34e96c3","entry":"encode","repo":"PatxiofromAlphensign/subword-nmt-mods","repo_kind":"listed","path":"subword_nmt/apply_bpe.py","file_url":"https://github.com/PatxiofromAlphensign/subword-nmt-mods/blob/HEAD/subword_nmt/apply_bpe.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"049cda35c34e96c3"}},{"code_sha256_prefix":"52bdb02fe848dfd2","entry":"extract_ngrams","repo":"PatxiofromAlphensign/subword-nmt-mods","repo_kind":"listed","path":"subword_nmt/chrF.py","file_url":"https://github.com/PatxiofromAlphensign/subword-nmt-mods/blob/HEAD/subword_nmt/chrF.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"52bdb02fe848dfd2"}},{"code_sha256_prefix":"9b27e34636c16aac","entry":"f1","repo":"PatxiofromAlphensign/subword-nmt-mods","repo_kind":"listed","path":"subword_nmt/chrF.py","file_url":"https://github.com/PatxiofromAlphensign/subword-nmt-mods/blob/HEAD/subword_nmt/chrF.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9b27e34636c16aac"}},{"code_sha256_prefix":"e985a436f19df6e2","entry":"get_correct","repo":"PatxiofromAlphensign/subword-nmt-mods","repo_kind":"listed","path":"subword_nmt/chrF.py","file_url":"https://github.com/PatxiofromAlphensign/subword-nmt-mods/blob/HEAD/subword_nmt/chrF.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e985a436f19df6e2"}},{"code_sha256_prefix":"e7b94766e4ee985e","entry":"get_stats","repo":"PatxiofromAlphensign/subword-nmt-mods","repo_kind":"listed","path":"subword_nmt/bpe_toy.py","file_url":"https://github.com/PatxiofromAlphensign/subword-nmt-mods/blob/HEAD/subword_nmt/bpe_toy.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7b94766e4ee985e"}},{"code_sha256_prefix":"7d923800aaefd70c","entry":"get_vocabulary","repo":"PatxiofromAlphensign/subword-nmt-mods","repo_kind":"listed","path":"subword_nmt/learn_bpe.py","file_url":"https://github.com/PatxiofromAlphensign/subword-nmt-mods/blob/HEAD/subword_nmt/learn_bpe.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7d923800aaefd70c"}},{"code_sha256_prefix":"6428a14ec7b62a2c","entry":"merge_vocab","repo":"PatxiofromAlphensign/subword-nmt-mods","repo_kind":"listed","path":"subword_nmt/bpe_toy.py","file_url":"https://github.com/PatxiofromAlphensign/subword-nmt-mods/blob/HEAD/subword_nmt/bpe_toy.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6428a14ec7b62a2c"}},{"code_sha256_prefix":"6a9c6917f21d9cd1","entry":"validate_ignore_ids","repo":"VKCOM/YouTokenToMe","repo_kind":"listed","path":"youtokentome/yttm_cli.py","file_url":"https://github.com/VKCOM/YouTokenToMe/blob/HEAD/youtokentome/yttm_cli.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6a9c6917f21d9cd1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}