Papers › LERT: A Linguistically-motivated Pre-trained Language Model
LERT: A Linguistically-motivated Pre-trained Language Model
Yiming Cui, Wanxiang Che, Shijin Wang, Ting Liu
Pre-trained Language Model (PLM) has become a representative foundation model in the natural language processing field. Most PLMs are trained with linguistic-agnostic pre-training tasks on the surface form of the text, such as the masked language model (MLM). To further empower the PLMs with richer linguistic features, in this paper, we aim to propose a simple but effective way to learn linguistic features for pre-trained language models. We propose LERT, a pre-trained language model that is trained on three types of linguistic features along with the original MLM pre-training task, using a linguistically-informed pre-training (LIP) strategy. We carried out extensive experiments on ten Chinese NLU tasks, and the experimental results show that LERT could bring significant improvements over various comparable baselines. Furthermore, we also conduct analytical experiments in various linguistic aspects, and the results prove that the design of LERT is valid and effective. Resources are available at https://github.com/ymcui/LERT
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
1 archive task tag without a task page not shown.
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Stock Market Prediction | Astock | Chinese Lert Large (News+Factors) | Accuray | 66.36 | #6 of 17 | Archive leaderboard | report |
| Stock Market Prediction | Astock | Chinese Lert Large (News+Factors) | F1-score | 66.16 | #6 of 17 | Archive leaderboard | report |
| Stock Market Prediction | Astock | Chinese Lert Large (News+Factors) | Precision | 66.40 | #6 of 17 | Archive leaderboard | report |
| Stock Market Prediction | Astock | Chinese Lert Large (News+Factors) | Recall | 66.69 | #6 of 17 | Archive leaderboard | report |
| Stock Market Prediction | Astock | Chinese Lert Large (News) | Accuray | 64.37 | #9 of 17 | Archive leaderboard | report |
| Stock Market Prediction | Astock | Chinese Lert Large (News) | F1-score | 64.30 | #9 of 17 | Archive leaderboard | report |
| Stock Market Prediction | Astock | Chinese Lert Large (News) | Precision | 64.34 | #9 of 17 | Archive leaderboard | report |
| Stock Market Prediction | Astock | Chinese Lert Large (News) | Recall | 64.31 | #9 of 17 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections