Papers › Bilingual Alignment Pre-Training for Zero-Shot Cross-Lingual Transfer

Bilingual Alignment Pre-Training for Zero-Shot Cross-Lingual Transfer

3 Jun 2021EMNLP (MRQA) 2021 11arXiv:2106.01732archive 2025-07-28

Ziqing Yang, Wentao Ma, Yiming Cui, Jiani Ye, Wanxiang Che, Shijin Wang

Multilingual pre-trained models have achieved remarkable performance on cross-lingual transfer learning. Some multilingual models such as mBERT, have been pre-trained on unlabeled corpora, therefore the embeddings of different languages in the models may not be aligned very well. In this paper, we aim to improve the zero-shot cross-lingual transfer performance by proposing a pre-training task named Word-Exchange Aligning Model (WEAM), which uses the statistical alignment information as the prior knowledge to guide cross-lingual word prediction. We evaluate our model on multilingual machine reading comprehension task MLQA and natural language interface task XNLI. The results show that WEAM can significantly improve the zero-shot performance.

PaperPDFConference PDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

Katherinaxxx/Paper-ReadingNotes mentioned on GitHubSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Cross-Lingual TransferLanguage ModellingMachine Reading ComprehensionReading ComprehensionTransfer LearningZero-Shot Cross-Lingual Transfer

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Methods

mBERT

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections