{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/incorporating-context-into-subword","title":"Incorporating Context into Subword Vocabularies","arxiv_id":"2210.07095","date":"2022-10-13","proceeding":null,"authors":["Shaked Yehezkel","Yuval Pinter"],"abstract":"Most current popular subword tokenizers are trained based on word frequency statistics over a corpus, without considering information about co-occurrence or context. Nevertheless, the resulting vocabularies are used in language models' highly contextualized settings. We present SaGe, a tokenizer that tailors subwords for their downstream use by baking in the contextualized signal at the vocabulary creation phase. We show that SaGe does a better job than current widespread tokenizers in keeping token contexts cohesive, while not incurring a large price in terms of encoding efficiency or domain robustness. SaGe improves performance on English GLUE classification tasks as well as on NER, and on Inference and NER in Turkish, demonstrating its robustness to language properties such as morphological exponence and agglutination.","url_abs":"https://arxiv.org/abs/2210.07095v2","url_pdf":"https://arxiv.org/pdf/2210.07095v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"incorporating-context-into-subword","repo_url":"https://github.com/melelbgu/sage","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"cg","task_name":"NER"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.07095","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.07095"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/melelbgu/sage","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2a8a9e315ef6767c","entry":"compute_window","repo":"melelbgu/sage","repo_kind":"official","path":"sage_v1/Python-Modules/Utils.py","file_url":"https://github.com/melelbgu/sage/blob/HEAD/sage_v1/Python-Modules/Utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2a8a9e315ef6767c"}},{"code_sha256_prefix":"bb4f6156fa7e1466","entry":"get_not_ablateable_tokens_list","repo":"melelbgu/sage","repo_kind":"official","path":"sage_v1/Python-Modules/Utils.py","file_url":"https://github.com/melelbgu/sage/blob/HEAD/sage_v1/Python-Modules/Utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bb4f6156fa7e1466"}},{"code_sha256_prefix":"3d7a8c27e8286ada","entry":"sigmoid","repo":"melelbgu/sage","repo_kind":"official","path":"sage_v1/Python-Modules/Utils.py","file_url":"https://github.com/melelbgu/sage/blob/HEAD/sage_v1/Python-Modules/Utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3d7a8c27e8286ada"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}