Datasets › StoryCloze

StoryCloze

Introduced by Nasrin Mostafazadeh et al. in A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories6 Apr 2016 archive 2025-07-28

Representation and learning of commonsense knowledge is one of the foundational problems in the quest to enable deep language understanding. This issue is particularly challenging for understanding casual and correlational relationships between events. While this topic has received a lot of interest in the NLP community, research has been hindered by the lack of a proper evaluation framework. This paper attempts to address this problem with a new framework for evaluating story understanding and script learning: the 'Story Cloze Test'. This test requires a system to choose the correct ending to a four-sentence story. We created a new corpus of ~50k five-sentence commonsense stories, ROCStories, to enable this evaluation. This corpus is unique in two ways: (1) it captures a rich set of causal and temporal commonsense relations between daily events, and (2) it is a high quality collection of everyday life stories that can also be used for story generation. Experimental evaluation shows that a host of baselines and state-of-the-art models based on shallow language understanding struggle to achieve a high score on the Story Cloze Test. We discuss these implications for script and story learning, and offer suggestions for deeper language understanding.

Benchmarks archive 2025-07-28

All 1 leaderboard whose dataset resolves to this page shown (sort by any header). "First row" is the archive's own first row at snapshot, in the archive's row order; nothing here re-ranks and metric direction is not asserted.

Syntology 1 newer paper was judged to report on this dataset's benchmarks so far (Claude Sonnet 4.5, reading their tables: a model's judgement, not a result; 0 placed): Question Answering · StoryCloze (1 judged, 0 placed). Drawn from 8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read).

First row (archive order)PaperCode
Question Answering StoryCloze BLOOMZ Accuracy 96.3 Crosslingual Generalization through Multitask Finetuning bigscience-workshop/xmtf 23 Compare

Papers archive 2025-07-28

papers with a benchmark row: 14 · with a code link: 10 · where Syntology ran a sample: 4 (3 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument) Syntology

Show: all papers with a benchmark rowonly where code ran (4 of 14 with a benchmark row: 3 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument)

Page 1 of 1: papers 1 to 14 of 14 with a leaderboard row on this dataset's benchmarks, newest first by the archive's date (ties by slug; undated papers last).

The archive's own "papers using this dataset" list was never published, so this is the benchmark-backed subset, not that list; the archive's count for this dataset is 51. The Syntology column is from Syntology's graph, stated per sample; it is not part of any archive number. A line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified” (C is a part of N, never taken away from it); the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. When the archive marks a repository official for the paper, the cell starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record.

PaperCodeResultsDateSamples run Syntology
The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning 2 1 23 May 2023 not harvested
Exploring the Benefits of Training Expert Language Models over Instruction Tuning 2 1 7 Feb 2023 official (archive's flag): 2 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (3 pointer-only for licence)
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot 6 5 2 Jan 2023 official (archive's flag): 2 ran · 9 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified (9 pointer-only for licence)
Crosslingual Generalization through Multitask Finetuning 1 1 3 Nov 2022 official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified
Knowledge-in-Context: Towards Knowledgeable Semi-Parametric Language Models 0 1 28 Oct 2022 not harvested
Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learners 1 1 6 Oct 2022 not harvested
Efficient Language Modeling with Sparse all-MLP 0 5 14 Mar 2022 not harvested
Finetuned Language Models Are Zero-Shot Learners 8 2 3 Sep 2021 official: harvested for another paper · 0 ran · 1 unverified
Language Models are Few-Shot Learners 67 1 28 May 2020 community repositories only · 45 ran (of which 0 constructed an object rather than computing a result; 37 with no instrument failure: 3 honoured, 2 violated, 32 with no contract checked; 8 where Syntology's instrument failed) · 20 unverified (7 pointer-only for licence)
Improving Machine Reading Comprehension with General Reading Strategies 1 1 31 Oct 2018 not harvested
Improving Language Understanding by Generative Pre-Training 13 1 11 Jun 2018 not harvested
A Simple and Effective Approach to the Story Cloze Test 0 1 15 Mar 2018 not harvested
Story Comprehension for Predicting What Happens Next 0 1 1 Sep 2017 not harvested
UNIMELB at SemEval-2016 Tasks 4A and 4B: An Ensemble of Neural Networks and a Word2Vec Based Model for Sentiment Classification 1 1 1 Jun 2016 not harvested

Dataset loaders archive 2025-07-28

No loader listed in the archive.

Tasks archive 2025-07-28

License archive 2025-07-28

No licence recorded in the archive. Absence here is not a statement about the dataset's terms.

Modalities archive 2025-07-28

No modality tagged.

Languages archive 2025-07-28

No language tagged.

Variants archive 2025-07-28

  • Story Cloze Test
  • StoryCloze

2 variant names, as the archive lists them.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections