{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2601-05776","title":"One Script Instead of Hundreds? On Pretraining Romanized Encoder Language Models","arxiv_id":"2601.05776","date":"2026-01-09","proceeding":null,"authors":["Benedikt Ebing","Lennart Keller","Goran Glavaš"],"abstract":"Exposing latent lexical overlap, script romanization has emerged as an effective strategy for improving cross-lingual transfer (XLT) in multilingual language models (mLMs). Most prior work, however, focused on setups that favor romanization the most: (1) transfer from high-resource Latin-script to low-resource non-Latin-script languages and/or (2) between genealogically closely related languages with different scripts. It thus remains unclear whether romanization is a good representation choice for pretraining general-purpose mLMs, or, more precisely, if information loss associated with romanization harms performance for high-resource languages. We address this gap by pretraining encoder LMs from scratch on both romanized and original texts for six typologically diverse high-resource languages, investigating two potential sources of degradation: (i) loss of script-specific information and (ii) negative cross-lingual interference from increased vocabulary overlap. Using two romanizers with different fidelity profiles, we observe negligible performance loss for languages with segmental scripts, whereas languages with morphosyllabic scripts (Chinese and Japanese) suffer degradation that higher-fidelity romanization mitigates but cannot fully recover. Importantly, comparing monolingual LMs with their mLM counterpart, we find no evidence that increased subword overlap induces negative interference. We further show that romanization improves encoding efficiency (i.e., fertility) for segmental scripts at a negligible performance cost.","url_abs":"https://arxiv.org/abs/2601.05776","url_pdf":"https://arxiv.org/pdf/2601.05776","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2601.05776"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/alexa/massive","reach":null},{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/AnswerDotAI/ModernBERT","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":14},"by_repo_kind":{"found_in_text":{"samples":14,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"eb04bca046263ad3","entry":"build_dataloader","repo":"AnswerDotAI/ModernBERT","repo_kind":"found_in_text","path":"src/convert_dataset.py","file_url":"https://github.com/AnswerDotAI/ModernBERT/blob/HEAD/src/convert_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eb04bca046263ad3"}},{"code_sha256_prefix":"926142a87d625531","entry":"build_hf_dataset","repo":"AnswerDotAI/ModernBERT","repo_kind":"found_in_text","path":"src/convert_dataset.py","file_url":"https://github.com/AnswerDotAI/ModernBERT/blob/HEAD/src/convert_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"926142a87d625531"}},{"code_sha256_prefix":"403818b9bcd41764","entry":"get_ba","repo":"AnswerDotAI/ModernBERT","repo_kind":"found_in_text","path":"download_artifacts_from_wandb.py","file_url":"https://github.com/AnswerDotAI/ModernBERT/blob/HEAD/download_artifacts_from_wandb.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"403818b9bcd41764"}},{"code_sha256_prefix":"d0c4790bb141a628","entry":"get_base_folder","repo":"AnswerDotAI/ModernBERT","repo_kind":"found_in_text","path":"download_artifacts_from_wandb.py","file_url":"https://github.com/AnswerDotAI/ModernBERT/blob/HEAD/download_artifacts_from_wandb.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d0c4790bb141a628"}},{"code_sha256_prefix":"04ebbc66bb4519f4","entry":"get_gpu_memory_usage","repo":"AnswerDotAI/ModernBERT","repo_kind":"found_in_text","path":"run_evals.py","file_url":"https://github.com/AnswerDotAI/ModernBERT/blob/HEAD/run_evals.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"04ebbc66bb4519f4"}},{"code_sha256_prefix":"31deca0db80e01a3","entry":"get_model_artifacts","repo":"AnswerDotAI/ModernBERT","repo_kind":"found_in_text","path":"download_artifacts_from_wandb.py","file_url":"https://github.com/AnswerDotAI/ModernBERT/blob/HEAD/download_artifacts_from_wandb.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"31deca0db80e01a3"}},{"code_sha256_prefix":"a7225f88ff3bc5fd","entry":"get_model_defaults","repo":"AnswerDotAI/ModernBERT","repo_kind":"found_in_text","path":"generate_eval_config.py","file_url":"https://github.com/AnswerDotAI/ModernBERT/blob/HEAD/generate_eval_config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a7225f88ff3bc5fd"}},{"code_sha256_prefix":"7571b888ddba7220","entry":"ordered_yaml_dump","repo":"AnswerDotAI/ModernBERT","repo_kind":"found_in_text","path":"generate_eval_config.py","file_url":"https://github.com/AnswerDotAI/ModernBERT/blob/HEAD/generate_eval_config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7571b888ddba7220"}},{"code_sha256_prefix":"b166dc11d0f54137","entry":"pad_input","repo":"AnswerDotAI/ModernBERT","repo_kind":"found_in_text","path":"src/bert_padding.py","file_url":"https://github.com/AnswerDotAI/ModernBERT/blob/HEAD/src/bert_padding.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b166dc11d0f54137"}},{"code_sha256_prefix":"c7c8aad3ab599318","entry":"rename_class","repo":"AnswerDotAI/ModernBERT","repo_kind":"found_in_text","path":"src/flex_bert.py","file_url":"https://github.com/AnswerDotAI/ModernBERT/blob/HEAD/src/flex_bert.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c7c8aad3ab599318"}},{"code_sha256_prefix":"c829869456a86404","entry":"run_job","repo":"AnswerDotAI/ModernBERT","repo_kind":"found_in_text","path":"run_evals.py","file_url":"https://github.com/AnswerDotAI/ModernBERT/blob/HEAD/run_evals.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c829869456a86404"}},{"code_sha256_prefix":"ee341a843ab02d5f","entry":"unpad_input","repo":"AnswerDotAI/ModernBERT","repo_kind":"found_in_text","path":"src/bert_padding.py","file_url":"https://github.com/AnswerDotAI/ModernBERT/blob/HEAD/src/bert_padding.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ee341a843ab02d5f"}},{"code_sha256_prefix":"e7df9e4a71a03583","entry":"unpad_input_only","repo":"AnswerDotAI/ModernBERT","repo_kind":"found_in_text","path":"src/bert_padding.py","file_url":"https://github.com/AnswerDotAI/ModernBERT/blob/HEAD/src/bert_padding.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e7df9e4a71a03583"}},{"code_sha256_prefix":"c9e2a83fdedb14c3","entry":"update_config","repo":"AnswerDotAI/ModernBERT","repo_kind":"found_in_text","path":"convert_to_hf.py","file_url":"https://github.com/AnswerDotAI/ModernBERT/blob/HEAD/convert_to_hf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c9e2a83fdedb14c3"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}