HellaSwag
Introduced by Rowan Zellers et al. in
HellaSwag: Can a Machine Really Finish Your Sentence? archive 2025-07-28
HellaSwag is a challenge dataset for evaluating commonsense NLI that is specially hard for state-of-the-art models, though its questions are trivial for humans (>95% accuracy).
Benchmarks archive 2025-07-28
All 6 leaderboards whose dataset resolves to this page shown (sort by any header). "First row" is the archive's own first row at snapshot, in the archive's row order; nothing here re-ranks and metric direction is not asserted.
Syntology 35 newer papers were judged to report on this dataset's benchmarks so far (Claude Sonnet 4.5, reading their tables: a model's judgement, not a result; 0 placed): Sentence Completion · HellaSwag (29 judged, 0 placed), parameter-efficient fine-tuning · HellaSwag (10 judged, 0 placed), Question Answering · HellaSwag (3 judged, 0 placed). Drawn from 8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read).
Papers archive 2025-07-28
papers with a benchmark row: 39 · with a code link: 32 · where Syntology ran a sample: 20 (18 with a run with no instrument failure, 2 where every run was a failure of Syntology's instrument) Syntology
Show:
all papers with a benchmark row only where code ran (20 of 39 with a benchmark row: 18 with a run with no instrument failure, 2 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 39 of 39 with a leaderboard row on this dataset's benchmarks, newest first by the archive's date (ties by slug; undated papers last).
The archive's own "papers using this dataset" list was never published, so this is the benchmark-backed subset, not that list; the archive's count for this dataset is 994. The Syntology column is from Syntology's graph, stated per sample; it is not part of any archive number. A line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified” (C is a part of N, never taken away from it); the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. When the archive marks a repository official for the paper, the cell starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. Syntology's record for this page last changed 2026-10-05 ; when this build read Syntology's graph is in the build record .
Paper Code Results Date Samples run Syntology
SHAKTI: A 2.5 Billion Parameter Small Language Model Optimized for Edge AI and Low-Resource Environments
0
1
15 Oct 2024
not harvested
GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs
1
1
27 Aug 2024
official (archive's flag): 2 ran · 5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 2 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (5 pointer-only for licence )
Mixture-of-Subspaces in Low-Rank Adaptation
1
1
16 Jun 2024
official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified (6 pointer-only for licence )
MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts
2
3
22 Apr 2024
official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 1 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 5 unverified
DoRA: Weight-Decomposed Low-Rank Adaptation
5
1
14 Feb 2024
official (archive's flag): 7 ran · 8 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 8 where Syntology's instrument failed) · 7 unverified (14 pointer-only for licence )
Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks
2
2
5 Jan 2024
not harvested
LLM in a flash: Efficient Large Language Model Inference with Limited Memory
0
2
12 Dec 2023
not harvested
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
35
2
1 Dec 2023
community repositories only · 28 ran (of which 7 constructed an object rather than computing a result; 20 with no instrument failure: 0 honoured, 0 violated, 20 with no contract checked; 8 where Syntology's instrument failed) · 34 unverified (29 pointer-only for licence )
The Falcon Series of Open Language Models
0
3
28 Nov 2023
not harvested
Mistral 7B
6
1
10 Oct 2023
official (archive's flag): 9 ran · 10 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (2 pointer-only for licence )
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
2
3
10 Oct 2023
official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified
Llama 2: Open Foundation and Fine-Tuned Chat Models
19
4
18 Jul 2023
community repositories only · 33 ran (of which 7 constructed an object rather than computing a result; 20 with no instrument failure: 1 honoured, 1 violated, 18 with no contract checked; 13 where Syntology's instrument failed) · 19 unverified (20 pointer-only for licence )
Stay on topic with Classifier-Free Guidance
0
3
30 Jun 2023
not harvested
The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning
2
1
23 May 2023
not harvested
PaLM 2 Technical Report
1
3
17 May 2023
not harvested
LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions
1
6
27 Apr 2023
not harvested
BloombergGPT: A Large Language Model for Finance
2
4
30 Mar 2023
not harvested
GPT-4 Technical Report
11
2
15 Mar 2023
community repositories only · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 2 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (1 pointer-only for licence )
LLaMA: Open and Efficient Foundation Language Models
57
4
27 Feb 2023
official: no sample here; runs from other or unrecorded repositories · 37 ran (of which 9 constructed an object rather than computing a result; 25 with no instrument failure: 3 honoured, 0 violated, 22 with no contract checked; 12 where Syntology's instrument failed) · 21 unverified (4 pointer-only for licence )
Exploring the Benefits of Training Expert Language Models over Instruction Tuning
2
1
7 Feb 2023
official (archive's flag): 2 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (3 pointer-only for licence )
Two is Better than Many? Binary Classification as an Effective Approach to Multi-Choice Question Answering
1
2
29 Oct 2022
official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 5 unverified
Knowledge-in-Context: Towards Knowledgeable Semi-Parametric Language Models
0
1
28 Oct 2022
not harvested
DiscoSense: Commonsense Reasoning with Discourse Connectives
1
2
22 Oct 2022
not harvested
Task Compass: Scaling Multi-task Pre-training with Task Prefix
1
3
12 Oct 2022
not harvested
Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learners
1
1
6 Oct 2022
not harvested
PaLM: Scaling Language Modeling with Pathways
7
3
5 Apr 2022
32 ran (of which 16 constructed an object rather than computing a result; 24 with no instrument failure: 2 honoured, 1 violated, 21 with no contract checked; 8 where Syntology's instrument failed) · 5 unverified
Training Compute-Optimal Large Language Models
2
1
29 Mar 2022
8 ran (of which 3 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 0 violated, 3 with no contract checked; 3 where Syntology's instrument failed) · 3 unverified (4 pointer-only for licence )
Efficient Language Modeling with Sparse all-MLP
0
5
14 Mar 2022
not harvested
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
2
2
28 Jan 2022
not harvested
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
3
1
8 Dec 2021
not harvested
Finetuned Language Models Are Zero-Shot Learners
8
2
3 Sep 2021
official: harvested for another paper · 0 ran · 1 unverified
LoRA: Low-Rank Adaptation of Large Language Models
74
1
17 Jun 2021
community repositories only · 51 ran (of which 19 constructed an object rather than computing a result; 43 with no instrument failure: 1 honoured, 0 violated, 42 with no contract checked; 8 where Syntology's instrument failed) · 33 unverified (30 pointer-only for licence )
UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark
1
1
24 Mar 2021
official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified
Muppet: Massive Multi-task Representations with Pre-Finetuning
2
1
26 Jan 2021
not harvested
DeBERTa: Decoding-enhanced BERT with Disentangled Attention
14
1
5 Jun 2020
official: no sample here; runs from other or unrecorded repositories · 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 2 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 9 unverified (3 pointer-only for licence )
Language Models are Few-Shot Learners
67
3
28 May 2020
community repositories only · 45 ran (of which 0 constructed an object rather than computing a result; 37 with no instrument failure: 3 honoured, 2 violated, 32 with no contract checked; 8 where Syntology's instrument failed) · 20 unverified (7 pointer-only for licence )
Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering
0
1
30 Oct 2019
not harvested
RoBERTa: A Robustly Optimized BERT Pretraining Approach
67
2
26 Jul 2019
community repositories only · 37 ran (of which 11 constructed an object rather than computing a result; 36 with no instrument failure: 2 honoured, 0 violated, 34 with no contract checked; 1 where Syntology's instrument failed) · 11 unverified (24 pointer-only for licence )
HellaSwag: Can a Machine Really Finish Your Sentence?
2
9
19 May 2019
4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (4 pointer-only for licence )
Dataset loaders archive 2025-07-28
8 loaders as listed in the archive; links are outbound and not re-checked here.
Tasks archive 2025-07-28
License archive 2025-07-28
MIT
Modalities archive 2025-07-28
No modality tagged.
Languages archive 2025-07-28
No language tagged.
Variants archive 2025-07-28
HellaSwag HellaSwag (10-Shot) HellaSwag TR
3 variant names, as the archive lists them.