Datasets › Localized Narratives
Localized Narratives
We propose Localized Narratives, a new form of multimodal image annotations connecting vision and language. We ask annotators to describe an image with their voice while simultaneously hovering their mouse over the region they are describing. Since the voice and the mouse pointer are synchronized, we can localize every single word in the description. This dense visual grounding takes the form of a mouse trace segment per word and is unique to our data. We annotated 849k images with Localized Narratives: the whole COCO, Flickr30k, and ADE20K datasets, and 671k images of Open Images, all of which we make publicly available. We provide an extensive analysis of these annotations showing they are diverse, accurate, and efficient to produce. We also demonstrate their utility on the application of controlled image captioning.
Benchmarks archive 2025-07-28
All 3 leaderboards whose dataset resolves to this page shown (sort by any header). "First row" is the archive's own first row at snapshot, in the archive's row order; nothing here re-ranks and metric direction is not asserted.
| First row (archive order) | Paper | Code | ||||
|---|---|---|---|---|---|---|
| Image Captioning | Localized Narratives | LoopCAG CIDEr 114.0 | Control Image Captioning Spatially and Temporally | — | 2 | Compare |
| Image Retrieval | Localized Narratives | OPT Text-to-image R@1 0.4196 | OPT: Omni-Perception Pre-Trainer for Cross-Modal... | mindspore-ai/models +1 | 1 | Compare |
| Text to Audio Retrieval | Localized Narratives | OPT Text-to-audio R@1 0.78 | OPT: Omni-Perception Pre-Trainer for Cross-Modal... | mindspore-ai/models +1 | 1 | Compare |
Papers archive 2025-07-28
3 shown of 3 papers with a leaderboard row on this dataset's benchmarks, newest first. The archive's own "papers using this dataset" list was never published, so this is the benchmark-backed subset; the archive's count for this dataset is 63. The Syntology column is from Syntology's graph (read 2026-09-24), stated per sample; it is not part of any archive number.
| Date | Samples run Syntology | |||
|---|---|---|---|---|
| Control Image Captioning Spatially and Temporally | 0 | 1 | 1 Aug 2021 | not harvested |
| OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation | 2 | 2 | 1 Jul 2021 | not harvested |
| Connecting Vision and Language with Localized Narratives | 1 | 1 | 6 Dec 2019 | not harvested |
Dataset loaders archive 2025-07-28
No loader listed in the archive.
Tasks archive 2025-07-28
License archive 2025-07-28
No licence recorded in the archive. Absence here is not a statement about the dataset's terms.
Modalities archive 2025-07-28
Languages archive 2025-07-28
Variants archive 2025-07-28
- Localized Narratives
1 variant name, as the archive lists them.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections