{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2601-05531","title":"DNATokenizer: A GPU-First Byte-to-Identifier Tokenizer for High-Throughput DNA Language Models","arxiv_id":"2601.05531","date":"2026-01-09","proceeding":null,"authors":["Eliatan Niktab","Hardip Patel"],"abstract":"Tokenization sits at the boundary between high-throughput genomic input and GPU compute, posing challenges in both algorithm design and system throughput. Overlapping k-mer tokenization can introduce information leakage under masked language modeling (MLM) and may degrade downstream accuracy. Single-nucleotide tokenization avoids leakage and preserves per-base fidelity, but it greatly increases sequence length for attention-based architectures. Non-overlapping k-mers and byte-pair encoding (BPE) provide compression and avoid leakage, at the cost of boundary sensitivity or reduced interpretability. Empirically, the choice of tokenization interacts strongly with model architecture and task requirements. At the system level, however, standard string tokenizers and host-bound vocabulary lookups dominate wall-clock time once inputs reach billions of bases, regardless of the tokenization algorithm. We present DNATok, a high-performance, GPU-first tokenization system that replaces general-purpose string processing with byte lookup table (LUT)-based identifier streaming and an overlapped host-to-device (H2D)/compute pipeline using pinned memory and architectural parallelism. DNATok is vocabulary-agnostic: it accelerates single-nucleotide, non-overlapping k-mer, and BPE tokenization, and integrates as a drop-in systems layer beneath genomic foundation models. DNATok achieves 84-95x higher encoding throughput than optimized Hugging Face baselines and up to 1.9x higher H2D throughput. End-to-end streaming reaches 1.27-1.84e8 tokens/s depending on configuration, effectively removing tokenization as a bottleneck for production-scale training and inference.","url_abs":"https://arxiv.org/abs/2601.05531","url_pdf":"https://arxiv.org/pdf/2601.05531","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2601.05531","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2601.05531"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/eniktab/DNAtok","reach":null}],"summary":{"ran":2,"ran_draft_wrong":7,"unverified":3},"by_repo_kind":{"found_in_text":{"samples":12,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fa0fc50546b7552b","entry":"_cache_dir","repo":"eniktab/DNAtok","repo_kind":"found_in_text","path":"src/gputok_bpe_backend.py","file_url":"https://github.com/eniktab/DNAtok/blob/HEAD/src/gputok_bpe_backend.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fa0fc50546b7552b"}},{"code_sha256_prefix":"9e53b54f9fad962f","entry":"_compile_normalizer","repo":"eniktab/DNAtok","repo_kind":"found_in_text","path":"src/gputok_bpe_backend.py","file_url":"https://github.com/eniktab/DNAtok/blob/HEAD/src/gputok_bpe_backend.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9e53b54f9fad962f"}},{"code_sha256_prefix":"2b8c490d0161e599","entry":"_compile_normalizer_step","repo":"eniktab/DNAtok","repo_kind":"found_in_text","path":"src/gputok_bpe_backend.py","file_url":"https://github.com/eniktab/DNAtok/blob/HEAD/src/gputok_bpe_backend.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2b8c490d0161e599"}},{"code_sha256_prefix":"262b6bfb2b7dca4c","entry":"_extract_merges","repo":"eniktab/DNAtok","repo_kind":"found_in_text","path":"src/gputok_bpe_backend.py","file_url":"https://github.com/eniktab/DNAtok/blob/HEAD/src/gputok_bpe_backend.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"262b6bfb2b7dca4c"}},{"code_sha256_prefix":"198acc38c6877af8","entry":"_find_gpu_tokenizer_dir","repo":"eniktab/DNAtok","repo_kind":"found_in_text","path":"src/gputok_bpe_backend.py","file_url":"https://github.com/eniktab/DNAtok/blob/HEAD/src/gputok_bpe_backend.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"198acc38c6877af8"}},{"code_sha256_prefix":"bbdc8db48f4f530a","entry":"_merges_cache_path","repo":"eniktab/DNAtok","repo_kind":"found_in_text","path":"src/gputok_bpe_backend.py","file_url":"https://github.com/eniktab/DNAtok/blob/HEAD/src/gputok_bpe_backend.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bbdc8db48f4f530a"}},{"code_sha256_prefix":"175d7a9d22624d9e","entry":"_read_tokenizer_json","repo":"eniktab/DNAtok","repo_kind":"found_in_text","path":"src/gputok_bpe_backend.py","file_url":"https://github.com/eniktab/DNAtok/blob/HEAD/src/gputok_bpe_backend.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"175d7a9d22624d9e"}},{"code_sha256_prefix":"e2c4909e68d62538","entry":"_reconstruct_gputok_vocab","repo":"eniktab/DNAtok","repo_kind":"found_in_text","path":"src/gputok_bpe_backend.py","file_url":"https://github.com/eniktab/DNAtok/blob/HEAD/src/gputok_bpe_backend.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e2c4909e68d62538"}},{"code_sha256_prefix":"7b41d718e62e6a81","entry":"_supported_tokenizer","repo":"eniktab/DNAtok","repo_kind":"found_in_text","path":"src/gputok_bpe_backend.py","file_url":"https://github.com/eniktab/DNAtok/blob/HEAD/src/gputok_bpe_backend.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7b41d718e62e6a81"}},{"code_sha256_prefix":"56beeae2c8124463","entry":"GPUTokBPEBackend","repo":"eniktab/DNAtok","repo_kind":"found_in_text","path":"src/gputok_bpe_backend.py","file_url":"https://github.com/eniktab/DNAtok/blob/HEAD/src/gputok_bpe_backend.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"56beeae2c8124463"}},{"code_sha256_prefix":"ce8d001e80e38fef","entry":"_build_gputok_instance","repo":"eniktab/DNAtok","repo_kind":"found_in_text","path":"src/gputok_bpe_backend.py","file_url":"https://github.com/eniktab/DNAtok/blob/HEAD/src/gputok_bpe_backend.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ce8d001e80e38fef"}},{"code_sha256_prefix":"f8e3942e2322bf8a","entry":"_load_gputok_extension","repo":"eniktab/DNAtok","repo_kind":"found_in_text","path":"src/gputok_bpe_backend.py","file_url":"https://github.com/eniktab/DNAtok/blob/HEAD/src/gputok_bpe_backend.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f8e3942e2322bf8a"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}