Papers › CamemBERT: a Tasty French Language Model
CamemBERT: a Tasty French Language Model
Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah, Benoît Sagot
Pretrained language models are now ubiquitous in Natural Language Processing. Despite their success, most available models have either been trained on English data or on the concatenation of data in multiple languages. This makes practical use of such models --in all languages except English-- very limited. In this paper, we investigate the feasibility of training monolingual Transformer-based language models for other languages, taking French as an example and evaluating our language models on part-of-speech tagging, dependency parsing, named entity recognition and natural language inference tasks. We show that the use of web crawled data is preferable to the use of Wikipedia data. More surprisingly, we show that a relatively small web crawled dataset (4GB) leads to results that are as good as those obtained using larger datasets (130+GB). Our best performing model CamemBERT reaches or improves the state of the art in all four downstream tasks.
In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.
Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
For agents, on Syntology's MCP service (how to connect):
get_citation_path(paper_1="1911.03894", paper_2="…")with another paper's arXiv id or titleget_concepts_for_paper(arxiv_id="1911.03894")
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Datasets
Introduced by this paper, per the archive.
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Dependency Parsing | French GSD | CamemBERT | LAS | 92.47 | #1 of 2 | Archive leaderboard | report |
| Dependency Parsing | French GSD | CamemBERT | UAS | 94.82 | #1 of 2 | Archive leaderboard | report |
| Dependency Parsing | ParTUT | CamemBERT | LAS | 92.9 | #1 of 2 | Archive leaderboard | report |
| Dependency Parsing | ParTUT | CamemBERT | UAS | 95.21 | #1 of 2 | Archive leaderboard | report |
| Dependency Parsing | Sequoia Treebank | CamemBERT | LAS | 94.39 | #1 of 2 | Archive leaderboard | report |
| Dependency Parsing | Sequoia Treebank | CamemBERT | UAS | 95.56 | #1 of 2 | Archive leaderboard | report |
| Dependency Parsing | Spoken Corpus | CamemBERT | LAS | 81.37 | #1 of 2 | Archive leaderboard | report |
| Dependency Parsing | Spoken Corpus | CamemBERT | UAS | 86.05 | #1 of 2 | Archive leaderboard | report |
| Named Entity Recognition (NER) | French Treebank | CamemBERT (subword masking) | F1 | 87.93 | #1 of 1 | Archive leaderboard | report |
| Named Entity Recognition (NER) | French Treebank | CamemBERT (subword masking) | Precision | 88.35 | #1 of 1 | Archive leaderboard | report |
| Named Entity Recognition (NER) | French Treebank | CamemBERT (subword masking) | Recall | 87.46 | #1 of 1 | Archive leaderboard | report |
| Natural Language Inference | XNLI French | CamemBERT (large) | Accuracy | 85.7 | #1 of 6 | Archive leaderboard | report |
| Natural Language Inference | XNLI French | CamemBERT (base) | Accuracy | 81.2 | #3 of 6 | Archive leaderboard | report |
| Part-Of-Speech Tagging | French GSD | CamemBERT | UPOS | 98.19 | #1 of 1 | Archive leaderboard | report |
| Part-Of-Speech Tagging | ParTUT | CamemBERT | UPOS | 97.63 | #1 of 1 | Archive leaderboard | report |
| Part-Of-Speech Tagging | Sequoia Treebank | CamemBERT | UPOS | 99.21 | #1 of 1 | Archive leaderboard | report |
| Part-Of-Speech Tagging | Spoken Corpus | CamemBERT | UPOS | 96.68 | #1 of 1 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections