Browse State-of-the-Art › Visual Grounding
Visual Grounding
299 papers with code · 4 benchmarks · 11 datasets archive 2025-07-28
Visual Grounding (VG) aims to locate the most relevant object or region in an image, based on a natural language query. The query can be a phrase, a sentence, or even a multi-round dialogue. There are three main challenges in VG:
- What is the main focus in a query?
- How to understand an image?
- How to locate an object?
Description from the archive archive 2025-07-28.
Benchmarks archive 2025-07-28
4 leaderboard tables shown for this task, 4 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.
Syntology 3 newer papers were judged to report on this task's benchmarks so far (Claude Sonnet 4.5, reading their tables: a model's judgement, not a result; 1 placed): RefCOCO+ testA (3 judged, 1 placed), RefCOCO+ test B (3 judged, 0 placed), RefCOCO+ val (3 judged, 1 placed). Drawn from 8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read).
| Dataset | Best model (first row in archive order) | Paper | Code | Syntology | Compare |
|---|---|---|---|---|---|
| RefCOCO+ testA (7 rows) | Florence-2-large-ft | Florence-2: Advancing a Unified Representation for a Variety of... | code | — | Compare |
| RefCOCO+ test B (6 rows) | Florence-2-large-ft | Florence-2: Advancing a Unified Representation for a Variety of... | code | — | Compare |
| RefCOCO+ val (6 rows) | Florence-2-large-ft | Florence-2: Advancing a Unified Representation for a Variety of... | code | — | Compare |
| RefCOCO testA (1 row) | HYDRA | HYDRA: A Hyper Agent for Dynamic Compositional Visual Reasoning | code | Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 2 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 3 unverified | Compare |
Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
11 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
3 subtasks in the archive's task tree.
Papers archive 2025-07-28
archive papers tagged: 571 · with a code link: 299 · where Syntology ran a sample: 111 (94 with a run with no instrument failure, 17 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (111 of 571 tagged: 94 with a run with no instrument failure, 17 where every run was a failure of Syntology's instrument)
Page 1 of 6: papers 1 to 100 of 571, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
6 Aug 2019 11 repositories listed Syntology 10 ran (of which 6 constructed an object rather than computing a result; 8 with no instrument failure: 0 honoured, 0 violated, 8 with no contract checked; 2 where Syntology's instrument failed) · 24 unverified (of 34 harvested samples) · 34 pointer-only (licence)We present ViLBERT (short for Vision-and-Language BERT), a model for learning task-agnostic joint representations of image content and natural language.
-
6 Jun 2016 10 repositories listedApproaches to multimodal pooling include element-wise product or sum, as well as concatenation of the visual and textual representations.
-
29 Mar 2023 7 repositories listed Syntology official (archive's flag): 4 ran · 8 ran (of which 2 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 1 violated, 5 with no contract checked; 2 where Syntology's instrument failed) · 4 unverified (of 12 harvested samples) · 12 pointer-only (licence)In this paper, we propose ViewRefer, a multi-view framework for 3D visual grounding exploring how to grasp the view knowledge from both text and 3D modalities.
-
26 Apr 2021 5 repositories listed Syntology community repositories only · 7 ran (of which 4 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 1 violated, 5 with no contract checked; 1 where Syntology's instrument failed) · 4 unverified (of 11 harvested samples)We also investigate the utility of our model as an object detector on a given label set when fine-tuned in a few-shot setting.
-
28 Dec 2024 4 repositories listed Syntology 13 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 6 where Syntology's instrument failed) · 2 unverified (of 15 harvested samples) · 9 pointer-only (licence)Finally, we outline the challenges confronting visual grounding and propose valuable directions for future research, which may serve as inspiration for subsequent researchers.
-
17 Oct 2023 4 repositories listed Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample)We present Set-of-Mark (SoM), a new visual prompting method, to unleash the visual grounding abilities of large multimodal models (LMMs), such as GPT-4V.
-
1 Feb 2023 4 repositories listed Syntology community repositories only · 17 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 0 honoured, 0 violated, 8 with no contract checked; 9 where Syntology's instrument failed) · 2 unverified (of 19 harvested samples) · 2 pointer-only (licence)In contrast to predominant paradigms of solely relying on sequence-to-sequence generation or encoder-based instance discrimination, mPLUG-2 introduces a multi-module composition network by sharing common universal…
-
7 Feb 2022 4 repositories listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)In this work, we pursue a unified paradigm for multimodal pretraining to break the scaffolds of complex task/modality-specific customization.
-
23 May 2025 3 repositories listedTo address this limitation, we introduce OrionBench, a benchmark designed to support the development of accurate object detection models for charts and HROs in infographics.
-
18 Jun 2024 3 repositories listed Syntology official (archive's flag): 4 ran · 11 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 1 violated, 3 with no contract checked; 7 where Syntology's instrument failed) · 2 unverified (of 13 harvested samples) · 13 pointer-only (licence)We introduce a new benchmark designed to advance the development of general-purpose, large-scale vision-language models for remote sensing images.
-
17 Jun 2024 3 repositories listed Syntology official (archive's flag): 1 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 3 unverified (of 5 harvested samples) · 1 pointer-only (licence)Scientific documents record research findings and valuable human knowledge, comprising a vast corpus of high-quality data.
-
27 Feb 2024 3 repositories listed Syntology official (archive's flag): 2 ran · 14 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 5 honoured, 1 violated, 0 with no contract checked; 8 where Syntology's instrument failed) · 3 unverified (of 17 harvested samples) · 8 pointer-only (licence)This paper presents ShapeLLM, the first 3D Multimodal Large Language Model (LLM) designed for embodied interaction, exploring a universal 3D object understanding with 3D point clouds and languages.
-
15 May 2023 3 repositories listedIn order to utilize vision and language pre-trained models to address the grounding problem, and reasonably take advantage of pseudo-labels, we propose CLIP-VG, a novel method that can conduct self-paced curriculum…
-
29 Sep 2022 3 repositories listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)3D visual grounding aims to find the object within point clouds mentioned by free-form natural language descriptions with rich semantic cues.
-
24 May 2022 3 repositories listedLarge-scale pretrained foundation models have been an emerging paradigm for building artificial intelligence (AI) systems, which can be quickly adapted to a wide range of downstream tasks.
-
30 Mar 2022 3 repositories listed Syntology official (archive's flag): 2 ran · 5 ran (of which 3 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified (of 7 harvested samples)To implement this idea, we propose Collaborative Glance-Gaze TransFormer (CoFormer) that consists of two modules: Glance transformer for activity classification and Gaze transformer for entity estimation.
-
30 Mar 2022 3 repositories listed Syntology community repositories only · 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 4 harvested samples) · 1 pointer-only (licence)In this paper, we propose a simple yet universal network termed SeqTR for visual grounding tasks, e. g., phrase localization, referring expression comprehension (REC) and segmentation (RES).
-
28 Mar 2022 3 repositories listedWe present a method for visually-grounded spoken term discovery.
-
10 Sep 2018 3 repositories listedWe compare our approach to an alternative system which extends the baseline with reinforcement learning.
-
27 Jun 2016 3 repositories listedVisual question answering (VQA) is an interesting learning setting for evaluating the abilities and shortcomings of current systems for image understanding.
-
12 Nov 2015 3 repositories listedWe propose a novel approach which learns grounding by reconstructing a given phrase using an attention mechanism, which can be either latent or optimized directly.
-
18 May 2025 2 repositories listedGraphical User Interface (GUI) agents have made substantial strides in understanding and executing user instructions across diverse platforms.
-
21 Jan 2025 2 repositories listedTo address this, we identify two critical sets of visual tokens that facilitate the transfer of visual information from the vision encoder to the LLM.
-
25 Nov 2024 2 repositories listedAdvances in multimodal pre-training have propelled object-level foundation models, such as Grounding DINO and Florence-2, in tasks like visual grounding and object detection.
-
11 Jun 2024 2 repositories listedGrounded Multimodal Named Entity Recognition (GMNER) task aims to identify named entities, entity types and their corresponding visual regions.
-
9 Jun 2024 2 repositories listedTo address this issue, we present F-LMM -- grounding frozen off-the-shelf LMMs in human-AI conversations -- a straightforward yet effective design based on the fact that word-pixel correspondences conducive to visual…
-
29 Mar 2024 2 repositories listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 4 harvested samples) · 1 pointer-only (licence)VHM is built on a large-scale remote sensing image-text dataset with rich-content captions (VersaD), and an honest instruction dataset comprising both factual and deceptive questions (HnstD).
-
21 Mar 2024 2 repositories listedToday's most accurate language models are trained on orders of magnitude more language data than human language learners receive - but with no supervision from other sensory modalities that play a crucial role in human…
-
23 Feb 2024 2 repositories listed Syntology official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 1 violated, 9 with no contract checked; 1 where Syntology's instrument failed) · 6 unverified (of 17 harvested samples) · 17 pointer-only (licence)Large Vision-Language Models (LVLMs) are susceptible to object hallucinations, an issue in which their generated text contains non-existent objects, greatly limiting their reliability and practicality.
-
15 Feb 2024 2 repositories listedGrounded Multimodal Named Entity Recognition (GMNER) is a nascent multimodal task that aims to identify named entities, entity types and their corresponding visual regions.
-
28 Dec 2023 2 repositories listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
4 Dec 2023 2 repositories listed Syntology official (archive's flag): 4 ran · 11 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 0 honoured, 0 violated, 9 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified (of 14 harvested samples)
-
12 Nov 2023 2 repositories listed
-
20 Oct 2023 2 repositories listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
14 Oct 2023 2 repositories listed Syntology 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 1 pointer-only (licence)
-
Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug 2023 2 repositories listed Syntology official: harvested for another paper · 0 ran · 2 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
22 Aug 2023 2 repositories listed
-
26 Jun 2023 2 repositories listed
-
19 May 2023 2 repositories listed Syntology official (archive's flag): 14 ran · 14 ran (of which 0 constructed an object rather than computing a result; 12 with no instrument failure: 3 honoured, 1 violated, 8 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified (of 16 harvested samples) · 1 pointer-only (licence)
-
22 Nov 2022 2 repositories listed Syntology official (archive's flag): 2 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified (of 6 harvested samples) · 6 pointer-only (licence)
-
24 Sep 2021 2 repositories listed Syntology official (archive's flag): 12 ran · 12 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 13 harvested samples) · 2 pointer-only (licence)
-
17 Apr 2021 2 repositories listed Syntology community repositories only · 18 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 8 where Syntology's instrument failed) · 2 unverified (of 20 harvested samples) · 8 pointer-only (licence)
-
14 Mar 2021 2 repositories listed Syntology 11 ran (of which 10 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 1 where Syntology's instrument failed) · 5 unverified (of 16 harvested samples) · 6 pointer-only (licence)
-
16 Feb 2021 2 repositories listed
-
23 Sep 2020 2 repositories listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
20 Nov 2019 2 repositories listed
-
18 Aug 2019 2 repositories listed Syntology official (archive's flag): 16 ran · 16 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 1 honoured, 0 violated, 13 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified (of 19 harvested samples) · 3 pointer-only (licence)
-
25 Feb 2019 2 repositories listed Syntology official (archive's flag): 1 ran · 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified (of 7 harvested samples) · 1 pointer-only (licence)
-
8 Jul 2025 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 3 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
8 Jul 2025 1 repository listed
-
30 Jun 2025 1 repository listed
-
26 Jun 2025 1 repository listed
-
11 Jun 2025 1 repository listed
-
26 May 2025 1 repository listed
-
23 May 2025 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
21 May 2025 1 repository listed Syntology official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 0 violated, 3 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified (of 10 harvested samples) · 3 pointer-only (licence)
-
21 May 2025 1 repository listed
-
17 May 2025 1 repository listed
-
16 May 2025 1 repository listed
-
13 May 2025 1 repository listed
-
9 May 2025 1 repository listed
-
7 May 2025 1 repository listed
-
13 Apr 2025 1 repository listed
-
10 Apr 2025 1 repository listed
-
3 Apr 2025 1 repository listed
-
31 Mar 2025 1 repository listed
-
29 Mar 2025 1 repository listed
-
19 Mar 2025 1 repository listed
-
18 Mar 2025 1 repository listed
-
17 Mar 2025 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)
-
17 Mar 2025 1 repository listed Syntology official (archive's flag): 5 ran · 5 ran (of which 4 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 1 where Syntology's instrument failed) · 5 unverified (of 10 harvested samples)
-
13 Mar 2025 1 repository listed
-
11 Mar 2025 1 repository listed
-
11 Mar 2025 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 2 harvested samples)
-
24 Feb 2025 1 repository listed
-
14 Feb 2025 1 repository listed
-
3 Feb 2025 1 repository listed Syntology official: no sample here; runs from other or unrecorded repositories · 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result (of 1 harvested sample)
-
1 Feb 2025 1 repository listed Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample)
-
17 Jan 2025 1 repository listed
-
16 Jan 2025 1 repository listed
-
12 Jan 2025 1 repository listed Syntology official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 6 where Syntology's instrument failed) · 4 unverified (of 13 harvested samples)
-
11 Jan 2025 1 repository listed
-
8 Jan 2025 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)
-
7 Jan 2025 1 repository listed Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 1 violated, 2 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 5 harvested samples) · 3 pointer-only (licence)
-
1 Jan 2025 1 repository listed
-
1 Jan 2025 1 repository listed
-
23 Dec 2024 1 repository listed
-
22 Dec 2024 1 repository listed
-
20 Dec 2024 1 repository listed Syntology 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
13 Dec 2024 1 repository listed Syntology official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 2 honoured, 0 violated, 6 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified (of 13 harvested samples) · 3 pointer-only (licence)
-
11 Dec 2024 1 repository listed
-
7 Dec 2024 1 repository listed
-
6 Dec 2024 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 8 unverified (of 9 harvested samples)
-
5 Dec 2024 1 repository listed
-
22 Nov 2024 1 repository listed
-
21 Nov 2024 1 repository listed
-
21 Nov 2024 1 repository listed
-
16 Nov 2024 1 repository listed
-
31 Oct 2024 1 repository listed
-
28 Oct 2024 1 repository listed
Syntology lines on 42 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections