Browse State-of-the-Art › Referring expression generation
Referring expression generation
26 papers with code · 2 benchmarks · 4 datasets archive 2025-07-28
Generate referring expressions
Description from the archive archive 2025-07-28.
Benchmarks archive 2025-07-28
2 leaderboard tables shown for this task, 2 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.
| Dataset | Best model (first row in archive order) | Paper | Code | Syntology | Compare |
|---|---|---|---|---|---|
| ColonINST-v1 (Seen) (17 rows) | ColonGPT (w/ LoRA, w/o extra data) | Frontiers in Intelligent Colonoscopy | code | Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified | Compare |
| ColonINST-v1 (Unseen) (17 rows) | ColonGPT (w/ LoRA, w/o extra data) | Frontiers in Intelligent Colonoscopy | code | Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified | Compare |
Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
4 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Papers archive 2025-07-28
archive papers tagged: 84 · with a code link: 26 · where Syntology ran a sample: 11 (9 with a run with no instrument failure, 2 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (11 of 84 tagged: 9 with a run with no instrument failure, 2 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 84 of 84, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
17 Apr 2023 13 repositories listed Syntology official (archive's flag): 4 ran · 16 ran (of which 6 constructed an object rather than computing a result; 8 with no instrument failure: 0 honoured, 1 violated, 7 with no contract checked; 8 where Syntology's instrument failed) · 35 unverified (of 51 harvested samples)Instruction tuning large language models (LLMs) using machine-generated instruction-following data has improved zero-shot capabilities on new tasks, but the idea is less explored in the multimodal field.
-
5 Oct 2023 9 repositories listed Syntology 6 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 3 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 3 unverified (of 9 harvested samples) · 8 pointer-only (licence)Large multimodal models (LMM) have recently shown encouraging progress with visual instruction tuning.
-
31 Jul 2016 4 repositories listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified (of 4 harvested samples) · 4 pointer-only (licence)Humans refer to objects in their environments all the time, especially in dialogue with other people.
-
27 Mar 2024 2 repositories listed Syntology official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 1 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 8 harvested samples) · 1 pointer-only (licence)We try to narrow the gap by mining the potential of VLMs for better performance and any-to-any workflow from three aspects, i. e., high-resolution visual tokens, high-quality data, and VLM-guided generation.
-
14 Oct 2023 2 repositories listed Syntology 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 1 pointer-only (licence)Motivated by this, we target to build a unified interface for completing many vision-language tasks including image description, visual question answering, and visual grounding, among others.
-
26 Jun 2023 2 repositories listedWe introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new capabilities of perceiving object descriptions (e. g., bounding boxes) and grounding text to the visual world.
-
22 Apr 2025 1 repository listedReferring Expression Generation (REG) is a core task for evaluating the pragmatic competence of vision-language systems, requiring not only accurate semantic grounding but also adherence to principles of cooperative…
-
22 Oct 2024 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified (of 6 harvested samples)Colonoscopy is currently one of the most sensitive screening methods for colorectal cancer.
-
4 Oct 2024 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)We introduce a task and dataset for referring expression generation and comprehension in multi-agent embodied environments.
-
26 Sep 2024 1 repository listedTo mitigate the tug-of-war problem of multi-modal multi-task optimization in MLLMs, recent advances primarily focus on improving the LLM components, while neglecting the connector that bridges the gap between modalities.
-
9 Sep 2024 1 repository listedSecond, we propose the use of discourse-aware comprehension guiding as part of a generate-and-rerank strategy through which candidate REs generated with our REG model are reranked based on their discourse-dependent…
-
18 Apr 2024 1 repository listedScene context is well known to facilitate humans' perception of visible objects.
-
25 Mar 2024 1 repository listed Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 4 where Syntology's instrument failed) · 1 unverified (of 8 harvested samples) · 8 pointer-only (licence)Multi-modal Large Language Models (MLLMs) have demonstrated their ability to perceive objects in still images, but their application in video-related tasks, such as object tracking, remains understudied.
-
5 Mar 2024 1 repository listed Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 3 unverified (of 7 harvested samples)Multimodal Large Language Model (MLLMs) leverages Large Language Models as a cognitive framework for diverse visual-language tasks.
-
18 Feb 2024 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 1 violated, 1 with no contract checked; 4 where Syntology's instrument failed) · 1 unverified (of 7 harvested samples) · 1 pointer-only (licence)Multimodal Large Language Models (MLLMs) have demonstrated notable capabilities in general visual understanding and reasoning tasks.
-
28 Dec 2023 1 repository listedWe present MobileVLM, a competent multimodal vision language model (MMVLM) targeted to run on mobile devices.
-
6 Nov 2023 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)In this work, we present Grounding LMM (GLaMM), the first model that can generate natural language responses seamlessly intertwined with corresponding object segmentation masks.
-
10 Sep 2023 1 repository listedWe address these concerns by introducing a collaborative image ranking task, a grounded agreement game we call "A Game Of Sorts".
-
19 Aug 2023 1 repository listedReferring Expression Generation (REG) aims to generate unambiguous Referring Expressions (REs) for objects in a visual scene, with a dual task of Referring Expression Comprehension (REC) to locate the referred object.
-
1 Jun 2023 1 repository listedIn this paper, we propose a cost-efficient approach for training a vision-language conversational assistant that can answer open-ended research questions of biomedical images.
-
24 May 2023 1 repository listedNLP tasks are typically defined extensionally through datasets containing example instantiations (e. g., pairs of image i and text t), but motivated intensionally through capabilities invoked in verbal descriptions of…
-
1 Aug 2021 1 repository listedThis study introduces an enriched version of the E2E dataset, one of the most popular language resources for data-to-text NLG.
-
22 Sep 2019 1 repository listedWe follow the step-by-step approach to neural data-to-text generation we proposed in Moryossef et al (2019), in which the generation process is divided into a text-planning stage followed by a plan-realization stage.
-
4 Sep 2019 1 repository listedReferring Expression Generation (REG) is the task of generating contextually appropriate references to entities.
-
1 Nov 2018 1 repository listedThis paper describes the enrichment of WebNLG corpus (Gardent et al., 2017a, b), with the aim to further extend its usefulness as a resource for evaluating common NLG tasks, including Discourse Ordering, Lexicalization…
-
21 May 2018 1 repository listedTraditionally, Referring Expression Generation (REG) models first decide on the form and then on the content of references to discourse entities in text, typically relying on features such as salience and grammatical…
-
Intrinsic Task-based Evaluation for Referring Expression Generation12 Feb 2024 0 repositories listedRecently, a human evaluation study of Referring Expression Generation (REG) models had an unexpected conclusion: on webnlg, Referring Expressions (REs) generated by the state-of-the-art neural models were not only…
-
Enhancing Visual Grounding and Generalization: A Multi-Task Cycle Training Approach for Vision-Language Models21 Nov 2023 0 repositories listedFurthermore, we contribute a VG dataset, especially with multi-tasks.
-
DisCLIP: Open-Vocabulary Referring Expression Generation30 May 2023 0 repositories listedReferring Expressions Generation (REG) aims to produce textual descriptions that unambiguously identifies specific objects within a visual scene.
-
Assessing Neural Referential Form Selectors on a Realistic Multilingual Dataset10 Oct 2022 0 repositories listedPrevious work on Neural Referring Expression Generation (REG) all uses WebNLG, an English dataset that has been shown to reflect a very limited range of referring expression (RE) use.
-
Referring Expressions with Rational Speech Act Framework: A Probabilistic Approach16 May 2022 0 repositories listed
-
Non-neural Models Matter: A Re-evaluation of Neural Referring Expression Generation Systems15 Mar 2022 0 repositories listed
-
Using Referring Expression Generation to Model Literary Style1 Dec 2021 0 repositories listed
-
Decoupling Pragmatics: Discriminative Decoding for Referring Expression Generation1 Oct 2021 0 repositories listed
-
What can Neural Referential Form Selectors Learn?15 Aug 2021 0 repositories listed
-
Perspective-corrected Spatial Referring Expression Generation for Human-Robot Interaction4 Apr 2021 0 repositories listed
-
Visual Question Answering based on Local-Scene-Aware Referring Expression Generation22 Jan 2021 0 repositories listed
-
CoNAN: A Complementary Neighboring-based Attention Network for Referring Expression Generation1 Dec 2020 0 repositories listed
-
Generating Quantified Referring Expressions through Attention-Driven Incremental Perception1 Dec 2020 0 repositories listed
-
Improving the Naturalness and Diversity of Referring Expression Generation models using Minimum Risk Training1 Dec 2020 0 repositories listed
-
OMEGA : A probabilistic approach to referring expression generation in a virtual environment1 Dec 2020 0 repositories listed
-
Referring to what you know and do not know: Making Referring Expression Generation Models Generalize To Unseen Entities1 Dec 2020 0 repositories listed
-
Lessons from Computational Modelling of Reference Production in Mandarin and English14 Nov 2020 0 repositories listed
-
Fuzzy Logic for Vagueness Management in Referring Expression Generation1 Sep 2020 0 repositories listed
-
Toward Forgetting-Sensitive Referring Expression Generationfor Integrated Robot Architectures16 Jul 2020 0 repositories listed
-
Informativity in Image Captions vs. Referring Expressions1 Jun 2020 0 repositories listed
-
MuDoCo: Corpus for Multidomain Coreference Resolution and Referring Expression Generation1 May 2020 0 repositories listed
-
A case study on context-bound referring expression generation1 Oct 2019 0 repositories listed
-
Augmenting Robot Knowledge Consultants with Distributed Short Term Memory26 Nov 2018 0 repositories listed
-
Adapting Descriptions of People to the Point of View of a Moving Observer1 Nov 2018 0 repositories listed
-
Decoding Strategies for Neural Referring Expression Generation1 Nov 2018 0 repositories listed
-
Statistical NLG for Generating the Content and Form of Referring Expressions1 Nov 2018 0 repositories listed
-
Generating Texts with Integer Linear Programming31 Oct 2018 0 repositories listed
-
An Incremental Iterated Response Model of Pragmatics30 Sep 2018 0 repositories listed
-
Meteorologists and Students: A resource for language grounding of geographical descriptors7 Sep 2018 0 repositories listed
-
Reference production in human-computer interaction: Issues for Corpus-based Referring Expression Generation1 May 2018 0 repositories listed
-
Referring Expression Generation in time-constrained communication1 May 2018 0 repositories listed
-
A Predictive Model for Notional Anaphora in English19 Apr 2018 0 repositories listed
-
Semi-automatic definite description annotation: a first report24 Dec 2017 0 repositories listed
-
Referring Expression Generation and Comprehension via Attributes1 Oct 2017 0 repositories listed
-
Exploring the Behavior of Classic REG Algorithms in the Description of Characters in 3D Images1 Sep 2017 0 repositories listed
-
G-TUNA: a corpus of referring expressions in German, including duration information1 Sep 2017 0 repositories listed
-
Improving the generation of personalised descriptions1 Sep 2017 0 repositories listed
-
Refer-iTTS: A System for Referring in Spoken Installments to Objects in Real-World Images1 Sep 2017 0 repositories listed
-
Referring Expression Generation under Uncertainty: Algorithm and Evaluation Framework1 Sep 2017 0 repositories listed
-
The WebNLG Challenge: Generating Text from RDF Data1 Sep 2017 0 repositories listed
-
1 Jul 2017 0 repositories listed
-
Obtaining referential word meanings from visual and distributional information: Experiments on object naming1 Jul 2017 0 repositories listed
-
Squib: Effects of Cognitive Effort on the Resolution of Overspecified Descriptions1 Jun 2017 0 repositories listed
-
Trainable Referring Expression Generation using Overspecification Preferences12 Apr 2017 0 repositories listed
-
Building Multimodal Simulations for Natural Language1 Apr 2017 0 repositories listed
-
An Empirical Approach for Modeling Fuzzy Geographical Descriptors30 Mar 2017 0 repositories listed
-
Comprehension-guided referring expressions12 Jan 2017 0 repositories listed
-
On the Robustness of Standalone Referring Expression Generation Algorithms Using RDF Data1 Sep 2016 0 repositories listed
-
Easy Things First: Installments Improve Referring Expression Generation for Objects in Photographs1 Aug 2016 0 repositories listed
-
The Methodius Corpus of Rhetorical Discourse Structures and Generated Texts1 May 2016 0 repositories listed
-
Learning Distributions over Logical Forms for Referring Expression Generation1 Oct 2013 0 repositories listed
-
Towards Situated Dialogue: Revisiting Referring Expression Generation1 Oct 2013 0 repositories listed
-
Justifying Corpus-Based Choices in Referring Expression Generation1 Sep 2013 0 repositories listed
-
Combining Referring Expression Generation and Surface Realization: A Corpus-Based Investigation of Architectures1 Aug 2013 0 repositories listed
-
Geração de Expressões de Referência usando Relações Espaciais (Referring Expression Generation Using Spatial Relations) [in Portuguese]1 Jan 2013 0 repositories listed
-
On The Feasibility of Open Domain Referring Expression Generation Using Large Scale Folksonomies1 Jun 2012 0 repositories listed
-
Learning Preferences for Referring Expression Generation: Effects of Domain, Language and Algorithm1 May 2012 0 repositories listed
-
SemScribe: Natural Language Generation for Medical Reports1 May 2012 0 repositories listed
Syntology lines on 11 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections