Papers › CamemBERT: a Tasty French Language Model
CamemBERT: a Tasty French Language Model
Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah, Benoît Sagot
Pretrained language models are now ubiquitous in Natural Language Processing. Despite their success, most available models have either been trained on English data or on the concatenation of data in multiple languages. This makes practical use of such models --in all languages except English-- very limited. In this paper, we investigate the feasibility of training monolingual Transformer-based language models for other languages, taking French as an example and evaluating our language models on part-of-speech tagging, dependency parsing, named entity recognition and natural language inference tasks. We show that the use of web crawled data is preferable to the use of Wikipedia data. More surprisingly, we show that a relatively small web crawled dataset (4GB) leads to results that are as good as those obtained using larger datasets (130+GB). Our best performing model CamemBERT reaches or improves the state of the art in all four downstream tasks.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Datasets
Introduced by this paper, per the archive.
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Dependency Parsing | French GSD | CamemBERT | LAS | 92.47 | #1 of 2 | Archive leaderboard | report |
| Dependency Parsing | French GSD | CamemBERT | UAS | 94.82 | #1 of 2 | Archive leaderboard | report |
| Dependency Parsing | ParTUT | CamemBERT | LAS | 92.9 | #1 of 2 | Archive leaderboard | report |
| Dependency Parsing | ParTUT | CamemBERT | UAS | 95.21 | #1 of 2 | Archive leaderboard | report |
| Dependency Parsing | Sequoia Treebank | CamemBERT | LAS | 94.39 | #1 of 2 | Archive leaderboard | report |
| Dependency Parsing | Sequoia Treebank | CamemBERT | UAS | 95.56 | #1 of 2 | Archive leaderboard | report |
| Dependency Parsing | Spoken Corpus | CamemBERT | LAS | 81.37 | #1 of 2 | Archive leaderboard | report |
| Dependency Parsing | Spoken Corpus | CamemBERT | UAS | 86.05 | #1 of 2 | Archive leaderboard | report |
| Named Entity Recognition (NER) | French Treebank | CamemBERT (subword masking) | F1 | 87.93 | #1 of 1 | Archive leaderboard | report |
| Named Entity Recognition (NER) | French Treebank | CamemBERT (subword masking) | Precision | 88.35 | #1 of 1 | Archive leaderboard | report |
| Named Entity Recognition (NER) | French Treebank | CamemBERT (subword masking) | Recall | 87.46 | #1 of 1 | Archive leaderboard | report |
| Natural Language Inference | XNLI French | CamemBERT (large) | Accuracy | 85.7 | #1 of 6 | Archive leaderboard | report |
| Natural Language Inference | XNLI French | CamemBERT (base) | Accuracy | 81.2 | #3 of 6 | Archive leaderboard | report |
| Part-Of-Speech Tagging | French GSD | CamemBERT | UPOS | 98.19 | #1 of 1 | Archive leaderboard | report |
| Part-Of-Speech Tagging | ParTUT | CamemBERT | UPOS | 97.63 | #1 of 1 | Archive leaderboard | report |
| Part-Of-Speech Tagging | Sequoia Treebank | CamemBERT | UPOS | 99.21 | #1 of 1 | Archive leaderboard | report |
| Part-Of-Speech Tagging | Spoken Corpus | CamemBERT | UPOS | 96.68 | #1 of 1 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections