Papers › PERT: Pre-training BERT with Permuted Language Model
PERT: Pre-training BERT with Permuted Language Model
Yiming Cui, Ziqing Yang, Ting Liu
Pre-trained Language Models (PLMs) have been widely used in various natural language processing (NLP) tasks, owing to their powerful text representations trained on large-scale corpora. In this paper, we propose a new PLM called PERT for natural language understanding (NLU). PERT is an auto-encoding model (like BERT) trained with Permuted Language Model (PerLM). The formulation of the proposed PerLM is straightforward. We permute a proportion of the input text, and the training objective is to predict the position of the original token. Moreover, we also apply whole word masking and N-gram masking to improve the performance of PERT. We carried out extensive experiments on both Chinese and English NLU benchmarks. The experimental results show that PERT can bring improvements over various comparable baselines on some of the tasks, while others are not. These results indicate that developing more diverse pre-training tasks is possible instead of masked language model variants. Several quantitative studies are carried out to better understand PERT, which might help design PLMs in the future. Resources are available: https://github.com/ymcui/PERT
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Stock Market Prediction | Astock | Chinese Pert Large (News+Factors) | Accuray | 67.37 | #4 of 17 | Archive leaderboard | report |
| Stock Market Prediction | Astock | Chinese Pert Large (News+Factors) | F1-score | 67.27 | #4 of 17 | Archive leaderboard | report |
| Stock Market Prediction | Astock | Chinese Pert Large (News+Factors) | Precision | 67.28 | #4 of 17 | Archive leaderboard | report |
| Stock Market Prediction | Astock | Chinese Pert Large (News+Factors) | Recall | 67.73 | #4 of 17 | Archive leaderboard | report |
| Stock Market Prediction | Astock | Chinese Pert Large (News) | Accuray | 65.09 | #8 of 17 | Archive leaderboard | report |
| Stock Market Prediction | Astock | Chinese Pert Large (News) | F1-score | 65.03 | #8 of 17 | Archive leaderboard | report |
| Stock Market Prediction | Astock | Chinese Pert Large (News) | Precision | 65.02 | #8 of 17 | Archive leaderboard | report |
| Stock Market Prediction | Astock | Chinese Pert Large (News) | Recall | 65.07 | #8 of 17 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections