{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/splintering-nonconcatenative-languages-for","title":"Splintering Nonconcatenative Languages for Better Tokenization","arxiv_id":"2503.14433","date":"2025-03-18","proceeding":null,"authors":["Bar Gazit","Shaltiel Shmidman","Avi Shmidman","Yuval Pinter"],"abstract":"Common subword tokenization algorithms like BPE and UnigramLM assume that text can be split into meaningful units by concatenative measures alone. This is not true for languages such as Hebrew and Arabic, where morphology is encoded in root-template patterns, or Malay and Georgian, where split affixes are common. We present SPLINTER, a pre-processing step which rearranges text into a linear form that better represents such nonconcatenative morphologies, enabling meaningful contiguous segments to be found by the tokenizer. We demonstrate SPLINTER's merit using both intrinsic measures evaluating token vocabularies in Hebrew, Arabic, and Malay; as well as on downstream tasks using BERT-architecture models trained for Hebrew.","url_abs":"https://arxiv.org/abs/2503.14433v1","url_pdf":"https://arxiv.org/pdf/2503.14433v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"splintering-nonconcatenative-languages-for","repo_url":"https://github.com/melelbgu/splintering","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"bpe","method_name":"BPE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.14433","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.14433"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/melelbgu/splintering","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7191118b793660b6","entry":"get_all_files_in_dir","repo":"melelbgu/splintering","repo_kind":"official","path":"src/save_dataset_as_text_file.py","file_url":"https://github.com/melelbgu/splintering/blob/HEAD/src/save_dataset_as_text_file.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7191118b793660b6"}},{"code_sha256_prefix":"f5e55af311e064c2","entry":"get_run_params","repo":"melelbgu/splintering","repo_kind":"official","path":"src/params.py","file_url":"https://github.com/melelbgu/splintering/blob/HEAD/src/params.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f5e55af311e064c2"}},{"code_sha256_prefix":"52bbb5a4a13e395a","entry":"split_to_separate_runs","repo":"melelbgu/splintering","repo_kind":"official","path":"src/params.py","file_url":"https://github.com/melelbgu/splintering/blob/HEAD/src/params.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"52bbb5a4a13e395a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}