{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/xlm-e-cross-lingual-language-model-pre","title":"XLM-E: Cross-lingual Language Model Pre-training via ELECTRA","arxiv_id":"2106.16138","date":"2021-06-30","proceeding":"ACL 2022 5","authors":["Zewen Chi","Shaohan Huang","Li Dong","Shuming Ma","Bo Zheng","Saksham Singhal","Payal Bajaj","Xia Song","Xian-Ling Mao","Heyan Huang","Furu Wei"],"abstract":"In this paper, we introduce ELECTRA-style tasks to cross-lingual language model pre-training. Specifically, we present two pre-training tasks, namely multilingual replaced token detection, and translation replaced token detection. Besides, we pretrain the model, named as XLM-E, on both multilingual and parallel corpora. Our model outperforms the baseline models on various cross-lingual understanding tasks with much less computation cost. Moreover, analysis shows that XLM-E tends to obtain better cross-lingual transferability.","url_abs":"https://arxiv.org/abs/2106.16138v2","url_pdf":"https://arxiv.org/pdf/2106.16138v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"xlm-e-cross-lingual-language-model-pre","repo_url":"https://github.com/microsoft/unilm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"xlm-e-cross-lingual-language-model-pre","repo_url":"https://github.com/CZWin32768/xnlg","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"xlm-e-cross-lingual-language-model-pre","repo_url":"https://github.com/Mind23-2/MindCode-151","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"translation","task_name":"Translation"},{"task_slug":"zero-shot-cross-lingual-transfer","task_name":"Zero-Shot Cross-Lingual Transfer"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/zero-shot-cross-lingual-transfer-on-xtreme","task":"Zero-Shot Cross-Lingual Transfer","dataset":"XTREME","model":"Turing ULR v6","rank_in_archive_order":1,"of":25,"metrics":{"Avg":"85.5","Question Answering":"77.1","Sentence Retrieval":"94.4","Sentence-pair Classification":"91.0","Structured Prediction":"83.8"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-cross-lingual-transfer-on-xtreme","task":"Zero-Shot Cross-Lingual Transfer","dataset":"XTREME","model":"Turing ULR v5","rank_in_archive_order":4,"of":25,"metrics":{"Avg":"84.5","Question Answering":"76.3","Sentence Retrieval":"93.7","Sentence-pair Classification":"90.3","Structured Prediction":"81.7"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2106.16138","atlas_url":"https://app.syntology.ai/?focus=2106.16138","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}