Datasets › RefCOCO

RefCOCO

Introduced by Sahar Kazemzadeh et al. in ReferItGame: Referring to Objects in Photographs of Natural Scenes1 Oct 2014 archive 2025-07-28

The RefCOCO dataset is a referring expression generation (REG) dataset used for tasks related to understanding natural language expressions that refer to specific objects in images. Here are the key details about RefCOCO:

  1. Collection Method: The dataset was collected using the ReferitGame, a two-player game. In this game, the first player views an image with a segmented target object and writes a natural language expression referring to that object. The second player sees only the image and the referring expression and must click on the corresponding object. If both players perform correctly, they earn points and switch roles; otherwise, they receive a new object and image for description.

  2. Dataset Variants: RefCOCO: Contains 142,209 refer expressions for 50,000 objects across 19,994 images. RefCOCO+: Includes 141,564 expressions for 49,856 objects in 19,992 images. RefCOCOg: This variant has 25,799 images, 95,010 referring expressions, and 49,822 object instances.

  3. Language and Restrictions: RefCOCO allows any type of language in the referring expressions. RefCOCO+ disallows location words in expressions to focus purely on appearance-based descriptions (e.g., "the man in the yellow polka-dotted shirt") rather than viewer-dependent descriptions (e.g., "the second man from the left").

These datasets serve as valuable resources for tasks like referring expression segmentation, comprehension, and visual grounding in computer vision research.

Benchmarks archive 2025-07-28

All 11 leaderboards whose dataset resolves to this page shown (sort by any header). "First row" is the archive's own first row at snapshot, in the archive's row order; nothing here re-ranks and metric direction is not asserted.

Syntology 10 newer papers were judged to report on this dataset's benchmarks so far (Claude Sonnet 4.5, reading their tables: a model's judgement, not a result; 1 placed): Referring Expression Segmentation · RefCoCo val (7 judged, 0 placed), Referring Expression Segmentation · RefCOCO+ testA (5 judged, 0 placed), Referring Expression Segmentation · RefCOCO+ test B (5 judged, 0 placed), Referring Expression Segmentation · RefCOCO testA (4 judged, 0 placed), Referring Expression Segmentation · RefCOCO testB (4 judged, 0 placed), Visual Grounding · RefCOCO+ testA (3 judged, 1 placed), Visual Grounding · RefCOCO+ test B (3 judged, 0 placed), Visual Grounding · RefCOCO+ val (3 judged, 1 placed), Referring Expression Segmentation · RefCOCO (1 judged, 0 placed). Drawn from 8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read).

First row (archive order)PaperCode
Referring Expression Segmentation RefCoCo val DeRIS-L Overall IoU 85.41 DeRIS: Decoupling Perception and Cognition for Enhanced... Dmmm1997/DeRIS 37 Compare
Referring Expression Segmentation RefCOCO+ val MLCD-Seg-7B Overall IoU 79.4 Multi-label Cluster Discrimination for Visual... deepglint/unicom 33 Compare
Referring Expression Segmentation RefCOCO+ testA HyperSeg Overall IoU 83.5 HyperSeg: Towards Universal Visual Segmentation with... congvvc/HyperSeg 30 Compare
Referring Expression Segmentation RefCOCO+ test B MLCD-Seg-7B Overall IoU 75.6 Multi-label Cluster Discrimination for Visual... deepglint/unicom 30 Compare
Referring Expression Segmentation RefCOCO testA DeRIS-L Overall IoU 86.49 DeRIS: Decoupling Perception and Cognition for Enhanced... Dmmm1997/DeRIS 13 Compare
Referring Expression Segmentation RefCOCO testB HyperSeg Overall IoU 83.4 HyperSeg: Towards Universal Visual Segmentation with... congvvc/HyperSeg 13 Compare
Visual Grounding RefCOCO+ testA Florence-2-large-ft Accuracy (%) 95.3 Florence-2: Advancing a Unified Representation for a... retkowsky/florence-2 7 Compare
Visual Grounding RefCOCO+ test B Florence-2-large-ft Accuracy (%) 92.0 Florence-2: Advancing a Unified Representation for a... retkowsky/florence-2 6 Compare
Visual Grounding RefCOCO+ val Florence-2-large-ft Accuracy (%) 93.4 Florence-2: Advancing a Unified Representation for a... retkowsky/florence-2 6 Compare
Referring Expression Segmentation RefCOCO DETRIS IoU 81.0 Densely Connected Parameter-Efficient Tuning for... jiaqihuang01/detris 4 Compare
Visual Grounding RefCOCO testA HYDRA IoU 61.7 HYDRA: A Hyper Agent for Dynamic Compositional Visual Reasoning ControlNet/HYDRA 1 Compare

Papers archive 2025-07-28

papers with a benchmark row: 43 · with a code link: 38 · where Syntology ran a sample: 25 (23 with a run with no instrument failure, 2 where every run was a failure of Syntology's instrument) Syntology

Show: all papers with a benchmark rowonly where code ran (25 of 43 with a benchmark row: 23 with a run with no instrument failure, 2 where every run was a failure of Syntology's instrument)

Page 1 of 1: papers 1 to 43 of 43 with a leaderboard row on this dataset's benchmarks, newest first by the archive's date (ties by slug; undated papers last).

The archive's own "papers using this dataset" list was never published, so this is the benchmark-backed subset, not that list; the archive's count for this dataset is 439. The Syntology column is from Syntology's graph, stated per sample; it is not part of any archive number. A line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified” (C is a part of N, never taken away from it); the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. When the archive marks a repository official for the paper, the cell starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record.

PaperCodeResultsDateSamples run Syntology
DeRIS: Decoupling Perception and Cognition for Enhanced Referring Image Segmentation through Loopback Synergy 1 6 2 Jul 2025 not harvested
SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectories 1 2 11 Mar 2025 not harvested
Densely Connected Parameter-Efficient Tuning for Referring Image Segmentation 1 7 15 Jan 2025 official (archive's flag): 13 ran · 13 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 9 where Syntology's instrument failed) · 4 unverified (9 pointer-only for licence)
Multi-task Visual Grounding with Coarse-to-Fine Consistency Constraints 1 6 12 Jan 2025 official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 6 where Syntology's instrument failed) · 4 unverified
MaskRIS: Semantic Distortion-aware Data Augmentation for Referring Image Segmentation 1 12 28 Nov 2024 not harvested
HyperSeg: Towards Universal Visual Segmentation with Large Language Model 1 6 26 Nov 2024 official (archive's flag): 13 ran · 13 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 4 honoured, 2 violated, 2 with no contract checked; 5 where Syntology's instrument failed) · 4 unverified (2 pointer-only for licence)
Multi-label Cluster Discrimination for Visual Representation Learning 1 6 24 Jul 2024 official (archive's flag): 7 ran · 7 ran (of which 7 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified; every one of the 7 samples that ran constructed an object rather than computing a result
SafaRi:Adaptive Sequence Transformer for Weakly Supervised Referring Expression Segmentation 0 6 2 Jul 2024 not harvested
EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Model 1 6 28 Jun 2024 official (archive's flag): 1 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (2 pointer-only for licence)
Vision-Aware Text Features in Referring Image Segmentation: From Object Understanding to Context Understanding 1 6 12 Apr 2024 not harvested
PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model 1 1 21 Mar 2024 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 4 unverified (2 pointer-only for licence)
HYDRA: A Hyper Agent for Dynamic Compositional Visual Reasoning 1 2 19 Mar 2024 official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 2 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 3 unverified
GROUNDHOG: Grounding Large Language Models to Holistic Segmentation 0 4 26 Feb 2024 not harvested
Mask Grounding for Referring Image Segmentation 1 6 19 Dec 2023 official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 8 where Syntology's instrument failed) · 3 unverified (13 pointer-only for licence)
General Object Foundation Model for Images and Videos at Scale 1 3 14 Dec 2023 official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 9 where Syntology's instrument failed) · 2 unverified (3 pointer-only for licence)
EVP: Enhanced Visual Perception using Inverse Multi-Attentive Feature Refinement and Regularized Image-Text Alignment 1 3 13 Dec 2023 not harvested
Universal Segmentation at Arbitrary Granularity with Language Instruction 2 7 4 Dec 2023 official (archive's flag): 14 ran · 14 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 12 where Syntology's instrument failed) · 2 unverified (7 pointer-only for licence)
Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks 1 3 10 Nov 2023 not harvested
Bridging Vision and Language Encoders: Parameter-Efficient Tuning for Referring Image Segmentation 1 2 21 Jul 2023 official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 6 where Syntology's instrument failed) · 5 unverified (8 pointer-only for licence)
Hierarchical Open-vocabulary Universal Image Segmentation 1 2 3 Jul 2023 official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (2 pointer-only for licence)
GRES: Generalized Referring Expression Segmentation 2 4 1 Jun 2023 official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified
Universal Instance Perception as Object Discovery and Retrieval 1 4 12 Mar 2023 official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (2 pointer-only for licence)
Unleashing Text-to-Image Diffusion Models for Visual Perception 2 1 3 Mar 2023 official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 2 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (3 pointer-only for licence)
PolyFormer: Referring Image Segmentation as Sequential Polygon Generation 1 8 14 Feb 2023 official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (6 pointer-only for licence)
mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video 4 3 1 Feb 2023 community repositories only · 17 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 0 honoured, 0 violated, 8 with no contract checked; 9 where Syntology's instrument failed) · 2 unverified (2 pointer-only for licence)
Toward Building General Foundation Models for Language, Vision, and Vision-Language Understanding Tasks 1 3 12 Jan 2023 not harvested
X²-VLM: All-In-One Pre-trained Model For Vision-Language Tasks 2 6 22 Nov 2022 official (archive's flag): 2 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified (6 pointer-only for licence)
VLT: Vision-Language Transformer and Query Generation for Referring Segmentation 1 4 28 Oct 2022 official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified
SeqTR: A Simple yet Universal Network for Visual Grounding 3 2 30 Mar 2022 community repositories only · 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (1 pointer-only for licence)
LAVT: Language-Aware Vision Transformer for Referring Image Segmentation 1 3 4 Dec 2021 not harvested
CRIS: CLIP-Driven Referring Image Segmentation 1 4 30 Nov 2021 official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 6 where Syntology's instrument failed) · 4 unverified (7 pointer-only for licence)
MaIL: A Unified Mask-Image-Language Trimodal Network for Referring Image Segmentation 0 4 21 Nov 2021 not harvested
Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts 1 3 16 Nov 2021 official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified
Vision-Language Transformer and Query Generation for Referring Segmentation 1 4 12 Aug 2021 not harvested
Referring Transformer: A One-step Approach to Multi-task Visual Grounding 1 1 6 Jun 2021 official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified
Comprehensive Multi-Modal Interactions for Referring Image Segmentation 1 4 21 Apr 2021 not harvested
Referring Image Segmentation via Cross-Modal Progressive Comprehension 1 4 1 Oct 2020 official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified
RefVOS: A Closer Look at Referring Expressions for Video Object Segmentation 2 5 1 Oct 2020 not harvested
Bi-Directional Relationship Inferring Network for Referring Image Segmentation 0 4 1 Jun 2020 not harvested
Referring Expression Object Segmentation with Caption-Aware Consistency 1 1 10 Oct 2019 official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 2 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified
See-Through-Text Grouping for Referring Image Segmentation 0 4 1 Oct 2019 not harvested
Cross-Modal Self-Attention Network for Referring Image Segmentation 1 4 9 Apr 2019 not harvested
MAttNet: Modular Attention Network for Referring Expression Comprehension 1 4 24 Jan 2018 not harvested

Dataset loaders archive 2025-07-28

2 loaders as listed in the archive; links are outbound and not re-checked here.

Tasks archive 2025-07-28

License archive 2025-07-28

Unknown

Modalities archive 2025-07-28

Languages archive 2025-07-28

No language tagged.

Variants archive 2025-07-28

  • RefCOCOg-val
  • RefCOCOg-test
  • RefCoco+
  • RefCOCO+ test B
  • RefCOCO+ testA
  • RefCOCO+ val
  • RefCOCO testB
  • RefCOCO testA
  • RefCoCo val
  • RefCOCO

10 variant names, as the archive lists them.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections