Papers › Language-agnostic BERT Sentence Embedding

Language-agnostic BERT Sentence Embedding

3 Jul 2020ACL 2022 5arXiv:2007.01852archive 2025-07-28

Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, Wei Wang

While BERT is an effective method for learning monolingual sentence embeddings for semantic similarity and embedding based transfer learning (Reimers and Gurevych, 2019), BERT based cross-lingual sentence embeddings have yet to be explored. We systematically investigate methods for learning multilingual sentence embeddings by combining the best methods for learning monolingual and cross-lingual representations including: masked language modeling (MLM), translation language modeling (TLM) (Conneau and Lample, 2019), dual encoder translation ranking (Guo et al., 2018), and additive margin softmax (Yang et al., 2019a). We show that introducing a pre-trained multilingual language model dramatically reduces the amount of parallel training data required to achieve good performance by 80%. Composing the best of these methods produces a model that achieves 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba, well above the 65.5% achieved by Artetxe and Schwenk (2019b), while still performing competitively on monolingual transfer learning benchmarks (Conneau and Kiela, 2018). Parallel data mined from CommonCrawl using our best model is shown to train competitive NMT models for en-zh and en-de. We publicly release our best multilingual sentence embedding model for 109+ languages at https://tfhub.dev/google/LaBSE.

Paper PDF Conference PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

FreddeFrallan/Multilingual-CLIP mentioned on GitHubpytorchMIT report

bojone/labse mentioned on GitHubtf report

jeongukjae/smaller-labse mentioned on GitHubtfApache-2.0 report

s-nlp/russe_detox_2022 mentioned on GitHubpytorch report

setu4993/convert-labse-tf-pt mentioned on GitHubpytorchMIT report

skoltech-nlp/russe_detox_2022 mentioned on GitHubpytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Language Modeling Language Modelling Masked Language Modeling Multilingual NLP NMT Retrieval Semantic Similarity Semantic Textual Similarity Sentence Sentence Embedding Sentence Embeddings Sentence-Embedding Text Retrieval Transfer Learning Translation

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Methods

Adam Attention Attention Dropout BERT Dense Connections Dropout Layer Normalization Linear Layer Linear Warmup With Linear Decay Multi-Head Attention Residual Connection Softmax Weight Decay WordPiece

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections