</> Code with Papers by Syntology

Papers › RAFT: A Real-World Few-Shot Text Classification Benchmark

RAFT: A Real-World Few-Shot Text Classification Benchmark

28 Sep 2021arXiv:2109.14076archive 2025-07-28

Neel Alex, Eli Lifland, Lewis Tunstall, Abhishek Thakur, Pegah Maham, C. Jess Riedel, Emmie Hine, Carolyn Ashurst, Paul Sedille, Alexis Carlier, Michael Noetel, Andreas Stuhlmüller

Large pre-trained language models have shown promise for few-shot learning, completing text-based tasks given only a few task-specific examples. Will models soon solve classification tasks that have so far been reserved for human research assistants? Existing benchmarks are not designed to measure progress in applied settings, and so don't directly answer this question. The RAFT benchmark (Real-world Annotated Few-shot Tasks) focuses on naturally occurring tasks and uses an evaluation setup that mirrors deployment. Baseline evaluations on RAFT reveal areas current techniques struggle with: reasoning over long texts and tasks with many classes. Human baselines show that some classification tasks are difficult for non-expert humans, reflecting that real-world value sometimes depends on domain expertise. Yet even non-expert human baseline F1 scores exceed GPT-3 by an average of 0.11. The RAFT datasets and leaderboard will track which model improvements translate into real-world benefits at https://raft.elicit.org .

PaperPDFCode

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

oughtinc/raft-baselines officialMIT report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

ClassificationFew-Shot LearningFew-Shot Text ClassificationText Classificationtext-classification

Datasets

Introduced by this paper, per the archive.

RAFT

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Few-Shot Text Classification RAFT Human (crowdsourced) Over 0.917 #2 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT Human (crowdsourced) ADE 0.830 #2 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT Human (crowdsourced) Avg 0.735 #2 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT Human (crowdsourced) B77 0.607 #2 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT Human (crowdsourced) NIS 0.857 #2 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT Human (crowdsourced) OSE 0.646 #2 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT Human (crowdsourced) SOT 0.908 #2 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT Human (crowdsourced) SRI 0.468 #2 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT Human (crowdsourced) TAI 0.609 #2 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT Human (crowdsourced) TC 0.897 #2 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT Human (crowdsourced) TEH 0.722 #2 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT Human (crowdsourced) ToS 0.627 #2 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-3 Over 0.937 #3 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-3 ADE 0.686 #3 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-3 Avg 0.627 #3 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-3 B77 0.299 #3 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-3 NIS 0.679 #3 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-3 OSE 0.431 #3 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-3 SOT 0.769 #3 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-3 SRI 0.516 #3 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-3 TAI 0.656 #3 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-3 TC 0.821 #3 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-3 TEH 0.526 #3 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-3 ToS 0.574 #3 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT AdaBoost Over 0.838 #4 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT AdaBoost ADE 0.543 #4 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT AdaBoost Avg 0.514 #4 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT AdaBoost B77 0.023 #4 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT AdaBoost NIS 0.626 #4 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT AdaBoost OSE 0.475 #4 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT AdaBoost SOT 0.455 #4 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT AdaBoost SRI 0.506 #4 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT AdaBoost TAI 0.556 #4 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT AdaBoost TC 0.625 #4 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT AdaBoost TEH 0.443 #4 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT AdaBoost ToS 0.560 #4 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-Neo Over 0.681 #5 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-Neo ADE 0.452 #5 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-Neo Avg 0.481 #5 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-Neo B77 0.149 #5 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-Neo NIS 0.408 #5 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-Neo OSE 0.343 #5 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-Neo SOT 0.406 #5 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-Neo SRI 0.493 #5 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-Neo TAI 0.605 #5 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-Neo TC 0.636 #5 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-Neo TEH 0.554 #5 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-Neo ToS 0.565 #5 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-2 Over 0.498 #6 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-2 ADE 0.600 #6 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-2 Avg 0.458 #6 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-2 B77 0.121 #6 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-2 NIS 0.561 #6 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-2 OSE 0.245 #6 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-2 SOT 0.380 #6 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-2 SRI 0.492 #6 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-2 TAI 0.612 #6 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-2 TC 0.723 #6 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-2 TEH 0.311 #6 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-2 ToS 0.498 #6 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT BART MNLI zero-shot Over 0.462 #7 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT BART MNLI zero-shot ADE 0.234 #7 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT BART MNLI zero-shot Avg 0.382 #7 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT BART MNLI zero-shot B77 0.332 #7 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT BART MNLI zero-shot NIS 0.615 #7 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT BART MNLI zero-shot OSE 0.360 #7 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT BART MNLI zero-shot SOT 0.644 #7 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT BART MNLI zero-shot SRI 0.026 #7 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT BART MNLI zero-shot TAI 0.469 #7 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT BART MNLI zero-shot TC 0.400 #7 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT BART MNLI zero-shot TEH 0.543 #7 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT BART MNLI zero-shot ToS 0.122 #7 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT Plurality-class Over 0.337 #8 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT Plurality-class ADE 0.446 #8 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT Plurality-class Avg 0.331 #8 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT Plurality-class B77 0.000 #8 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT Plurality-class NIS 0.353 #8 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT Plurality-class OSE 0.164 #8 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT Plurality-class SOT 0.271 #8 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT Plurality-class SRI 0.493 #8 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT Plurality-class TAI 0.344 #8 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT Plurality-class TC 0.391 #8 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT Plurality-class TEH 0.366 #8 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT Plurality-class ToS 0.471 #8 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-3 zero-shot Over 0.378 #9 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-3 zero-shot ADE 0.163 #9 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-3 zero-shot Avg 0.292 #9 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-3 zero-shot B77 0.000 #9 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-3 zero-shot NIS 0.572 #9 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-3 zero-shot OSE 0.323 #9 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-3 zero-shot SOT 0.628 #9 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-3 zero-shot SRI 0.027 #9 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-3 zero-shot TAI 0.362 #9 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-3 zero-shot TC 0.290 #9 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-3 zero-shot TEH 0.303 #9 of 9 Archive leaderboard report
Few-Shot Text Classification RAFT GPT-3 zero-shot ToS 0.164 #9 of 9 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

AdamAttentionAttention DropoutBPECosine AnnealingDense ConnectionsDropoutGPT-3Layer NormalizationLinear LayerLinear Warmup With Cosine AnnealingMulti-Head AttentionResidual ConnectionSoftmaxWeight Decay

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections