Papers › mGPT: Few-Shot Learners Go Multilingual

mGPT: Few-Shot Learners Go Multilingual

15 Apr 2022arXiv:2204.07580archive 2025-07-28

Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, Tatiana Shavrina

Recent studies report that autoregressive language models can successfully solve many NLP tasks via zero- and few-shot learning paradigms, which opens up new possibilities for using the pre-trained language models. This paper introduces two autoregressive GPT-like models with 1.3 billion and 13 billion parameters trained on 60 languages from 25 language families using Wikipedia and Colossal Clean Crawled Corpus. We reproduce the GPT-3 architecture using GPT-2 sources and the sparse attention mechanism; Deepspeed and Megatron frameworks allow us to parallelize the training and inference steps effectively. The resulting models show performance on par with the recently released XGLM models by Facebook, covering more languages and enhancing NLP possibilities for low resource languages of CIS countries and Russian small nations. We detail the motivation for the choices of the architecture design, thoroughly describe the data preparation pipeline, and train five small versions of the model to choose the most optimal multilingual tokenization strategy. We measure the model perplexity in all covered languages and evaluate it on the wide spectre of multilingual tasks, including classification, generative, sequence labeling and knowledge probing. The models were evaluated with the zero-shot and few-shot methods. Furthermore, we compared the classification tasks with the state-of-the-art multilingual model XGLM. source code and the mGPT XL model are publicly released.

Paper PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

ai-forever/mgpt officialmentioned in papermentioned on GitHubpytorchNOASSERTION report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Cross-Lingual Natural Language Inference Cross-Lingual Paraphrase Identification Cross-Lingual Transfer Few-Shot Learning Few-shot NER Knowledge Probing Natural Language Inference Part-Of-Speech Tagging

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Cross-Lingual Natural Language Inference	XNLI	mGPT	Accuracy	40.6	#5 of 5	Archive leaderboard	report
Cross-Lingual Transfer	XCOPA	mGPT	Accuracy	55.5	#6 of 6	Archive leaderboard	report
Few-shot NER	XGLUE	mGPT	Avg F1	0.85	#1 of 1	Archive leaderboard	report
Natural Language Inference	XWINO	mGPT	Accuracy	0.562	#1 of 1	Archive leaderboard	report
Part-Of-Speech Tagging	XGLUE	mGPT	Avg. F1	0.56	#1 of 1	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Adam Attention Attention Dropout BPE Cosine Annealing Dense Connections Discriminative Fine-Tuning Dropout GPT-2 GPT-3 Layer Normalization Linear Layer Linear Warmup With Cosine Annealing Multi-Head Attention Residual Connection Softmax Weight Decay

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections