Papers › Toward Efficient Language Model Pretraining and Downstream Adaptation via...
Toward Efficient Language Model Pretraining and Downstream Adaptation via Self-Evolution: A Case Study on SuperGLUE
Qihuang Zhong, Liang Ding, Yibing Zhan, Yu Qiao, Yonggang Wen, Li Shen, Juhua Liu, Baosheng Yu, Bo Du, Yixin Chen, Xinbo Gao, Chunyan Miao, Xiaoou Tang, DaCheng Tao
This technical report briefly describes our JDExplore d-team's Vega v2 submission on the SuperGLUE leaderboard. SuperGLUE is more challenging than the widely used general language understanding evaluation (GLUE) benchmark, containing eight difficult language understanding tasks, including question answering, natural language inference, word sense disambiguation, coreference resolution, and reasoning. [Method] Instead of arbitrarily increasing the size of a pretrained language model (PLM), our aim is to 1) fully extract knowledge from the input pretraining data given a certain parameter budget, e.g., 6B, and 2) effectively transfer this knowledge to downstream tasks. To achieve goal 1), we propose self-evolution learning for PLMs to wisely predict the informative tokens that should be masked, and supervise the masked language modeling (MLM) process with rectified smooth labels. For goal 2), we leverage the prompt transfer technique to improve the low-resource tasks by transferring the knowledge from the foundation model and related downstream tasks to the target task. [Results] According to our submission record (Oct. 2022), with our optimized pretraining and fine-tuning strategies, our 6B Vega method achieved new state-of-the-art performance on 4/8 tasks, sitting atop the SuperGLUE leaderboard on Oct. 8, 2022, with an average score of 91.3.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
No code repository is listed for this paper in the archive or in Syntology's graph.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Common Sense Reasoning | ReCoRD | Turing NLR v5 XXL 5.4B (fine-tuned) | EM | 95.9 | #1 of 45 | Archive leaderboard | report |
| Common Sense Reasoning | ReCoRD | Turing NLR v5 XXL 5.4B (fine-tuned) | F1 | 96.4 | #1 of 45 | Archive leaderboard | report |
| Common Sense Reasoning | ReCoRD | Vega v2 6B (fine-tuned) | EM | 93.9 | #5 of 45 | Archive leaderboard | report |
| Common Sense Reasoning | ReCoRD | Vega v2 6B (fine-tuned) | F1 | 94.4 | #5 of 45 | Archive leaderboard | report |
| Coreference Resolution | Winograd Schema Challenge | Vega v2 6B (KD-based prompt transfer) | Accuracy | 98.6 | #2 of 82 | Archive leaderboard | report |
| Coreference Resolution | Winograd Schema Challenge | Turing NLR v5 XXL 5.4B (fine-tuned) | Accuracy | 97.3 | #4 of 82 | Archive leaderboard | report |
| Natural Language Inference | CommitmentBank | Vega v2 6B (KD-based prompt transfer) | Accuracy | 99.2 | #2 of 20 | Archive leaderboard | report |
| Natural Language Inference | CommitmentBank | Vega v2 6B (KD-based prompt transfer) | F1 | 98.6 | #2 of 20 | Archive leaderboard | report |
| Natural Language Inference | CommitmentBank | Turing NLR v5 XXL 5.4B (fine-tuned) | Accuracy | 97.6 | #5 of 20 | Archive leaderboard | report |
| Natural Language Inference | CommitmentBank | Turing NLR v5 XXL 5.4B (fine-tuned) | F1 | 95.9 | #5 of 20 | Archive leaderboard | report |
| Natural Language Inference | RTE | Vega v2 6B (KD-based prompt transfer) | Accuracy | 96% | #1 of 90 | Archive leaderboard | report |
| Natural Language Inference | RTE | Turing NLR v5 XXL 5.4B (fine-tuned) | Accuracy | 94.1% | #3 of 90 | Archive leaderboard | report |
| Question Answering | BoolQ | Turing NLR v5 XXL 5.4B (fine-tuned) | Accuracy | 92 | #4 of 65 | Archive leaderboard | report |
| Question Answering | BoolQ | Vega v2 6B (fine-tuned) | Accuracy | 90.5 | #8 of 65 | Archive leaderboard | report |
| Question Answering | COPA | Vega v2 6B (KD-based prompt transfer) | Accuracy | 99.4 | #2 of 60 | Archive leaderboard | report |
| Question Answering | COPA | Turing NLR v5 XXL 5.4B (fine-tuned) | Accuracy | 98.2 | #6 of 60 | Archive leaderboard | report |
| Question Answering | MultiRC | Turing NLR v5 XXL 5.4B (fine-tuned) | EM | 63 | #3 of 30 | Archive leaderboard | report |
| Question Answering | MultiRC | Turing NLR v5 XXL 5.4B (fine-tuned) | F1 | 88.4 | #3 of 30 | Archive leaderboard | report |
| Question Answering | MultiRC | Vega v2 6B (fine-tuned) | EM | 62.4 | #5 of 30 | Archive leaderboard | report |
| Question Answering | MultiRC | Vega v2 6B (fine-tuned) | F1 | 88.2 | #5 of 30 | Archive leaderboard | report |
| Word Sense Disambiguation | Words in Context | Vega v2 6B (fine-tuned) | Accuracy | 77.4 | #5 of 37 | Archive leaderboard | report |
| Word Sense Disambiguation | Words in Context | Turing NLR v5 XXL 5.4B (fine-tuned) | Accuracy | 77.1 | #7 of 37 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections