{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-the-cross-lingual-transferability-of","title":"On the Cross-lingual Transferability of Monolingual Representations","arxiv_id":"1910.11856","date":"2019-10-25","proceeding":"ACL 2020 6","authors":["Mikel Artetxe","Sebastian Ruder","Dani Yogatama"],"abstract":"State-of-the-art unsupervised multilingual models (e.g., multilingual BERT) have been shown to generalize in a zero-shot cross-lingual setting. This generalization ability has been attributed to the use of a shared subword vocabulary and joint training across multiple languages giving rise to deep multilingual abstractions. We evaluate this hypothesis by designing an alternative approach that transfers a monolingual model to new languages at the lexical level. More concretely, we first train a transformer-based masked language model on one language, and transfer it to a new language by learning a new embedding matrix with the same masked language modeling objective, freezing parameters of all other layers. This approach does not rely on a shared vocabulary or joint training. However, we show that it is competitive with multilingual BERT on standard cross-lingual classification benchmarks and on a new Cross-lingual Question Answering Dataset (XQuAD). Our results contradict common beliefs of the basis of the generalization ability of multilingual models and suggest that deep monolingual models learn some abstractions that generalize across languages. We also release XQuAD as a more comprehensive cross-lingual benchmark, which comprises 240 paragraphs and 1190 question-answer pairs from SQuAD v1.1 translated into ten languages by professional translators.","url_abs":"https://arxiv.org/abs/1910.11856v3","url_pdf":"https://arxiv.org/pdf/1910.11856v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-the-cross-lingual-transferability-of","repo_url":"https://github.com/deepmind/xquad","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"on-the-cross-lingual-transferability-of","repo_url":"https://github.com/alon-albalak/xor-covid","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"on-the-cross-lingual-transferability-of","repo_url":"https://github.com/ccasimiro88/TranslateAlignRetrieve","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"on-the-cross-lingual-transferability-of","repo_url":"https://github.com/google-deepmind/xquad","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"on-the-cross-lingual-transferability-of","repo_url":"https://github.com/lersouza/cross-lingual-transfer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"on-the-cross-lingual-transferability-of","repo_url":"https://github.com/lmarent/TranslateAlignRetrieve","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"on-the-cross-lingual-transferability-of","repo_url":"https://github.com/somiltg/xquad","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"cross-lingual-question-answering","task_name":"Cross-Lingual Question Answering"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"masked-language-modeling","task_name":"Masked Language Modeling"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[{"slug":"xquad","name":"XQuAD","full_name":"XQuAD"}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1910.11856","atlas_url":"https://app.syntology.ai/?focus=1910.11856","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1910.11856"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google-deepmind/xquad","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ccasimiro88/TranslateAlignRetrieve","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lersouza/cross-lingual-transfer","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lmarent/TranslateAlignRetrieve","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deepmind/xquad","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/somiltg/xquad","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/alon-albalak/xor-covid","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"unverified":1},"by_repo_kind":{"listed":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9430683808935722","entry":"checkDate","repo":"alon-albalak/xor-covid","repo_kind":"listed","path":"testDates.py","file_url":"https://github.com/alon-albalak/xor-covid/blob/HEAD/testDates.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9430683808935722"}},{"code_sha256_prefix":"9a1240d74f429b49","entry":"checkLanguage","repo":"alon-albalak/xor-covid","repo_kind":"listed","path":"covid_qa_demo.py","file_url":"https://github.com/alon-albalak/xor-covid/blob/HEAD/covid_qa_demo.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a1240d74f429b49"}},{"code_sha256_prefix":"d354687d5c6880f7","entry":"convert_COUGH_to_retrieval_dataset","repo":"alon-albalak/xor-covid","repo_kind":"listed","path":"splitCOUGH.py","file_url":"https://github.com/alon-albalak/xor-covid/blob/HEAD/splitCOUGH.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d354687d5c6880f7"}},{"code_sha256_prefix":"5d121d6b67c4adda","entry":"dateInRange","repo":"alon-albalak/xor-covid","repo_kind":"listed","path":"testDates.py","file_url":"https://github.com/alon-albalak/xor-covid/blob/HEAD/testDates.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5d121d6b67c4adda"}},{"code_sha256_prefix":"03c8e71d5877a0d5","entry":"load_csv_to_list","repo":"alon-albalak/xor-covid","repo_kind":"listed","path":"splitCOUGH.py","file_url":"https://github.com/alon-albalak/xor-covid/blob/HEAD/splitCOUGH.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"03c8e71d5877a0d5"}},{"code_sha256_prefix":"507efbaa4928f16d","entry":"loss_dense","repo":"alon-albalak/xor-covid","repo_kind":"listed","path":"criterions.py","file_url":"https://github.com/alon-albalak/xor-covid/blob/HEAD/criterions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"507efbaa4928f16d"}},{"code_sha256_prefix":"81605ae475dd6fab","entry":"chunk_text","repo":"alon-albalak/xor-covid","repo_kind":"listed","path":"internalCORD_utils.py","file_url":"https://github.com/alon-albalak/xor-covid/blob/HEAD/internalCORD_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81605ae475dd6fab"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}