Papers › Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learners

Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learners

6 Oct 2022arXiv:2210.02969archive 2025-07-28

Seonghyeon Ye, Doyoung Kim, Joel Jang, Joongbo Shin, Minjoon Seo

Meta-training, which fine-tunes the language model (LM) on various downstream tasks by maximizing the likelihood of the target label given the task instruction and input instance, has improved the zero-shot task generalization performance. However, meta-trained LMs still struggle to generalize to challenging tasks containing novel labels unseen during meta-training. In this paper, we propose Flipped Learning, an alternative method of meta-training which trains the LM to generate the task instruction given the input instance and label. During inference, the LM trained with Flipped Learning, referred to as Flipped, selects the label option that is most likely to generate the task instruction. On 14 tasks of the BIG-bench benchmark, the 11B-sized Flipped outperforms zero-shot T0-11B and even a 16 times larger 3-shot GPT-3 (175B) on average by 8.4% and 9.7% points, respectively. Flipped gives particularly large improvements on tasks with unseen labels, outperforming T0-11B by up to +20% average F1 score. This indicates that the strong task generalization of Flipped comes from improved generalization to novel labels. We release our code at https://github.com/seonghyeonye/Flipped-Learning.

Paper PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

seonghyeonye/flipped-learning officialmentioned in papermentioned on GitHubpytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Common Sense Reasoning Coreference Resolution Language Modeling Language Modelling Natural Language Inference Natural Language Inference (Zero-Shot)Question Answering Sentence Completion Word Sense Disambiguation

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Common Sense Reasoning	WinoGrande	Flipped-3B	Accuracy	58.56	#55 of 77	Archive leaderboard	report
Coreference Resolution	Winograd Schema Challenge	Flipped-3B	Accuracy	58.37	#62 of 82	Archive leaderboard	report
Natural Language Inference	ANLI test	Flipped-3B	A1	39.99	#12 of 25	Archive leaderboard	report
Natural Language Inference	ANLI test	Flipped-3B	A2	37.05	#12 of 25	Archive leaderboard	report
Natural Language Inference	ANLI test	Flipped-3B	A3	37.73	#12 of 25	Archive leaderboard	report
Natural Language Inference	RTE	Flipped-3B	Accuracy	71.05	#52 of 90	Archive leaderboard	report
Question Answering	COPA	Flipped-3B	Accuracy	89.88	#19 of 60	Archive leaderboard	report
Question Answering	StoryCloze	Flipped-3B	Accuracy	95.88	#2 of 23	Archive leaderboard	report
Sentence Completion	HellaSwag	Flipped-3B	Accuracy	41.6	#74 of 89	Archive leaderboard	report
Word Sense Disambiguation	Words in Context	Flipped-3B	Accuracy	50.42	#33 of 37	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Adam Attention Attention Dropout BPE Cosine Annealing Dense Connections Dropout GPT-3 Layer Normalization Linear Layer Linear Warmup With Cosine Annealing Multi-Head Attention Residual Connection Softmax Weight Decay

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections