</> Code with Papers by Syntology

Papers › Task Compass: Scaling Multi-task Pre-training with Task Prefix

Task Compass: Scaling Multi-task Pre-training with Task Prefix

12 Oct 2022arXiv:2210.06277archive 2025-07-28

Zhuosheng Zhang, Shuohang Wang, Yichong Xu, Yuwei Fang, Wenhao Yu, Yang Liu, Hai Zhao, Chenguang Zhu, Michael Zeng

Leveraging task-aware annotated data as supervised signals to assist with self-supervised learning on large-scale unlabeled data has become a new trend in pre-training language models. Existing studies show that multi-task learning with large-scale supervised tasks suffers from negative effects across tasks. To tackle the challenge, we propose a task prefix guided multi-task pre-training framework to explore the relationships among tasks. We conduct extensive experiments on 40 datasets, which show that our model can not only serve as the strong foundation backbone for a wide range of tasks but also be feasible as a probing tool for analyzing task relationships. The task relationships reflected by the prefixes align transfer learning performance between tasks. They also suggest directions for data augmentation with complementary tasks, which help our model achieve human-parity results on commonsense reasoning leaderboards. Code is available at https://github.com/cooelf/CompassMTL

PaperPDFCode

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

cooelf/compassmtl officialmentioned in papermentioned on GitHubtf report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Common Sense ReasoningData AugmentationMulti-Task LearningQuestion AnsweringSelf-Supervised LearningSentence CompletionTransfer Learning

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Common Sense Reasoning WinoGrande CompassMTL 567M with Tailor Accuracy 90.5 #3 of 77 Archive leaderboard report
Common Sense Reasoning WinoGrande CompassMTL 567M Accuracy 89.6 #4 of 77 Archive leaderboard report
Common Sense Reasoning WinoGrande ExDeBERTa 567M Accuracy 87 #8 of 77 Archive leaderboard report
Question Answering PIQA CompassMTL 567M with Tailor Accuracy 88.3 #3 of 67 Archive leaderboard report
Question Answering PIQA CompassMTL 567M Accuracy 87.3 #6 of 67 Archive leaderboard report
Question Answering PIQA ExDeBERTa 567M Accuracy 85.5 #10 of 67 Archive leaderboard report
Question Answering SIQA CompassMTL 567M with Tailor Accuracy 82.2 #3 of 24 Archive leaderboard report
Question Answering SIQA CompassMTL 567M Accuracy 81.7 #4 of 24 Archive leaderboard report
Question Answering SIQA ExDeBERTa 567M Accuracy 79.6 #9 of 24 Archive leaderboard report
Sentence Completion HellaSwag CompassMTL 567M with Tailor Accuracy 96.1 #1 of 89 Archive leaderboard report
Sentence Completion HellaSwag CompassMTL 567M Accuracy 95.6 #2 of 89 Archive leaderboard report
Sentence Completion HellaSwag ExDeBERTa 567M Accuracy 83.6 #31 of 89 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

ALIGN

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections