Browse State-of-the-Art › Visual Dialog
Visual Dialog
56 papers with code · 8 benchmarks · 9 datasets archive 2025-07-28
Visual Dialog requires an AI agent to hold a meaningful dialog with humans in natural, conversational language about visual content. Specifically, given an image, a dialog history, and a follow-up question about the image, the task is to answer the question.
Description from the archive archive 2025-07-28.
Benchmarks archive 2025-07-28
8 leaderboard tables shown for this task, 8 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.
Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
9 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Parent tasks archive 2025-07-28
Papers archive 2025-07-28
archive papers tagged: 118 · with a code link: 56 · where Syntology ran a sample: 11 (10 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (11 of 118 tagged: 10 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument)
Page 1 of 2: papers 1 to 100 of 118, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
26 Nov 2016 11 repositories listed Syntology community repositories only · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)We introduce the task of Visual Dialog, which requires an AI agent to hold a meaningful dialog with humans in natural, conversational language about visual content.
-
31 May 2016 9 repositories listed Syntology community repositories only · 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 1 where Syntology's instrument failed) · 3 unverified (of 7 harvested samples) · 1 pointer-only (licence)In addition, our model reasons about the question (and consequently the image via the co-attention mechanism) in a hierarchical fashion via a novel 1-dimensional convolution neural networks (CNN).
-
20 Mar 2017 7 repositories listed Syntology 4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 4 harvested samples) · 1 pointer-only (licence)Specifically, we pose a cooperative 'image guessing' game between two agents -- Qbot and Abot -- who communicate in natural language dialog so that Qbot can select an unseen image from a lineup of images.
-
1 Jun 2018 4 repositories listedScene-aware dialog systems will be able to have conversations with users about the objects and events around them.
-
27 Oct 2021 3 repositories listed Syntology official (archive's flag): 1 ran · 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result (of 1 harvested sample) · 1 pointer-only (licence)To study and quantify this concern, we introduce the perceptual score, a metric that assesses the degree to which a model relies on the different subsets of the input features, i. e., modalities.
-
27 Mar 2024 2 repositories listed Syntology official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 1 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 8 harvested samples) · 1 pointer-only (licence)We try to narrow the gap by mining the potential of VLMs for better performance and any-to-any workflow from three aspects, i. e., high-resolution visual tokens, high-quality data, and VLM-guided generation.
-
25 May 2022 2 repositories listedAs a result, GST scales the amount of training data up to an order of magnitude that of VisDial (1.
-
16 Nov 2020 2 repositories listedIn this paper, we focus on the LED task -- providing a strong baseline model with detailed ablations characterizing both dataset biases and the importance of various modeling choices.
-
8 May 2020 2 repositories listedVisual Dialog involves "understanding" the dialog history (what has been discussed previously) and the current question (what is asked), in addition to grounding information in the image, to generate the correct…
-
5 Dec 2019 2 repositories listed Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified (of 6 harvested samples) · 2 pointer-only (licence)Next, we find that additional finetuning using "dense" annotations in VisDial leads to even higher NDCG -- more than 10% over our base model -- but hurts MRR -- more than 17% below our base model!
-
25 Feb 2019 2 repositories listed Syntology official (archive's flag): 1 ran · 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified (of 7 harvested samples) · 1 pointer-only (licence)Specifically, REFER module learns latent relationships between a given question and a dialog history by employing a self-attention mechanism.
-
16 Dec 2018 2 repositories listedWe characterise some of the quirks and shortcomings in the exploration of Visual Dialogue - a sequential question-answering task where the questions and corresponding answers are related through given visual stimuli.
-
27 May 2024 1 repository listed Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 6 where Syntology's instrument failed) · 2 unverified (of 9 harvested samples) · 9 pointer-only (licence)Video Anomaly Detection (VAD) systems can autonomously monitor and identify disturbances, reducing the need for manual labor and associated costs.
-
10 Sep 2023 1 repository listedWe address these concerns by introducing a collaborative image ranking task, a grounded agreement game we call "A Game Of Sorts".
-
24 May 2023 1 repository listed Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample)It utilizes a combination of several fundamental experts to accommodate multiple dialogue-related tasks and can be pre-trained using limited dialogue and extensive non-dialogue multi-modal data.
-
23 Nov 2022 1 repository listedPrior work performs the standard likelihood training for answer generation on the positive instances (involving correct answers).
-
15 Sep 2022 1 repository listedWe introduce LAVIS, an open-source deep learning library for LAnguage-VISion research and applications.
-
8 Jul 2022 1 repository listedTo tackle these challenges we present a new object-centric framework for video dialog that supports neural reasoning dubbed COST - which stands for Conversation about Objects in Space-Time.
-
29 May 2022 1 repository listedIn this paper, we propose VD-PCR, a novel framework to improve Visual Dialog understanding with Pronoun Coreference Resolution in both implicit and explicit ways.
-
16 Mar 2022 1 repository listedVisual dialog has witnessed great progress after introducing various vision-oriented goals into the conversation, especially such as GuessWhich and GuessWhat, where the only image is visible by either and both of the…
-
7 Dec 2021 1 repository listedOur model ranks second in the official evaluation on the object coreference resolution task with an F1 score of 73.
-
6 Sep 2021 1 repository listedTo enhance VD Questioner: 1) we propose a Related entity enhanced Questioner (ReeQ) that generates questions under the guidance of related entities and learns entity-based questioning strategy from human dialogs; 2) we…
-
1 Aug 2021 1 repository listedIn this work, we formulate a visual dialog as an information flow in which each piece of information is encoded with the joint visual-linguistic representation of a single dialog round.
-
24 May 2021 1 repository listedMost existing work for Guesser encode the dialog history as a whole and train the Guesser models from scratch on the GuessWhat?!
-
15 Apr 2021 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)However, the NDCG metric favors the usually applicable uncertain answers such as `I don't know.
-
1 Oct 2020 1 repository listedIn this paper, we propose an Answer-Driven Visual State Estimator (ADVSE) to impose the effects of different answers on visual states.
-
2 Aug 2020 1 repository listedIP based SeqDialN is our baseline with a simple 2-layer LSTM design that achieves decent performance.
-
24 Jul 2020 1 repository listedCan we develop visually grounded dialog agents that can efficiently adapt to new tasks without forgetting how to talk to people?
-
29 Apr 2020 1 repository listedTo resolve the visual dialog task, a high-level understanding of various multimodal inputs (e. g., question, dialog history, and image) is required.
-
28 Apr 2020 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 1 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 9 unverified (of 11 harvested samples)By contrast, in this work, we propose VD-BERT, a simple yet effective framework of unified vision-dialog Transformer that leverages the pretrained BERT language models for Visual Dialog tasks.
-
14 Apr 2020 1 repository listed
-
5 Apr 2020 1 repository listed
-
18 Dec 2019 1 repository listed
-
1 Dec 2019 1 repository listed
-
26 Nov 2019 1 repository listed
-
24 Nov 2019 1 repository listed
-
18 Nov 2019 1 repository listed
-
17 Nov 2019 1 repository listed
-
12 Nov 2019 1 repository listed
-
31 Oct 2019 1 repository listed
-
23 Sep 2019 1 repository listed
-
11 Apr 2019 1 repository listed
-
11 Apr 2019 1 repository listed
-
7 Mar 2019 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
6 Mar 2019 1 repository listed
-
22 Feb 2019 1 repository listed
-
6 Dec 2018 1 repository listed
-
6 Sep 2018 1 repository listed
-
3 Aug 2018 1 repository listed
-
2 Jul 2018 1 repository listed
-
17 May 2018 1 repository listed
-
1 May 2018 1 repository listed
-
12 Feb 2018 1 repository listed
-
4 Dec 2017 1 repository listed
-
5 Jun 2017 1 repository listed
-
18 Apr 2017 1 repository listed
-
V²Dial: Unification of Video and Visual Dialog via Multimodal Experts3 Mar 2025 0 repositories listed
-
V^2Dial: Unification of Video and Visual Dialog via Multimodal Experts1 Jan 2025 0 repositories listed
-
Enhancing Visual Dialog State Tracking through Iterative Object-Entity Alignment in Multi-Round Conversations13 Aug 2024 0 repositories listed
-
ICCV23 Visual-Dialog Emotion Explanation Challenge: SEU_309 Team Technical Report13 Jul 2024 0 repositories listed
-
FlexCap: Describe Anything in Images in Controllable Detail18 Mar 2024 0 repositories listed
-
𝕍𝔻-𝔾ℝ: Boosting 𝕍isual 𝔻ialog with Cascaded Spatial-Temporal Multi-Modal 𝔾ℝaphs25 Oct 2023 0 repositories listed
-
Affective Visual Dialog: A Large-Scale Benchmark for Emotional Reasoning Based on Visually Grounded Conversations30 Aug 2023 0 repositories listed
-
A survey on knowledge-enhanced multimodal learning19 Nov 2022 0 repositories listed
-
Knowledge Transfer with Visual Prompt in multi-modal Dialogue Understanding and Generation1 Oct 2022 0 repositories listed
-
Adversarial Robustness of Visual Dialog6 Jul 2022 0 repositories listed
-
ENRICH4ALL: A First Luxembourgish BERT Model for a Multilingual Chatbot1 Jun 2022 0 repositories listed
-
UTC: A Unified Transformer with Inter-Task Contrastive Learning for Visual Dialog1 May 2022 0 repositories listed
-
Improving Cross-Modal Understanding in Visual Dialog via Contrastive Learning15 Apr 2022 0 repositories listed
-
Reasoning with Multi-Structure Commonsense Knowledge in Visual Dialog10 Apr 2022 0 repositories listed
-
Modeling Coreference Relations in Visual Dialog6 Mar 2022 0 repositories listed
-
VU-BERT: A Unified framework for Visual Dialog22 Feb 2022 0 repositories listed
-
Discourse Analysis for Evaluating Coherence in Video Paragraph Captions17 Jan 2022 0 repositories listed
-
How to Fool Systems and Humans in Visually Grounded Interaction: A Case Study on Adversarial Attacks on Visual Dialog16 Jan 2022 0 repositories listed
-
Enriching Language Models with Visually-grounded Word Vectors and the Lancaster Sensorimotor Norms1 Nov 2021 0 repositories listed
-
Region under Discussion for visual dialog1 Nov 2021 0 repositories listed
-
ViDA-MAN: Visual Dialog with Digital Humans26 Oct 2021 0 repositories listed
-
Evaluating and Improving Interactions with Hazy Oracles19 Oct 2021 0 repositories listed
-
The Impact of Answers in Referential Visual Dialog1 Oct 2021 0 repositories listed
-
Variational Disentangled Attention for Regularized Visual Dialog29 Sep 2021 0 repositories listed
-
GoG: Relation-aware Graph-over-Graph Network for Visual Dialog17 Sep 2021 0 repositories listed
-
Learning to Ground Visual Objects for Visual Dialog13 Sep 2021 0 repositories listed
-
Visual-Textual Alignment for Graph Inference in Visual Dialog1 Dec 2020 0 repositories listed
-
Reasoning Over History: Context Aware Visual Dialog2 Nov 2020 0 repositories listed
-
2 Oct 2020 0 repositories listed
-
Effective questions in referential visual dialogue1 Jul 2020 0 repositories listed
-
Towards Visual Dialog for Radiology1 Jul 2020 0 repositories listed
-
ORD: Object Relationship Discovery for Visual Dialogue Generation15 Jun 2020 0 repositories listed
-
Modality-Balanced Models for Visual Dialogue17 Jan 2020 0 repositories listed
-
Ensemble based discriminative models for Visual Dialog Challenge 201815 Jan 2020 0 repositories listed
-
Vision and Language: from Visual Perception to Content Creation26 Dec 2019 0 repositories listed
-
Video Dialog via Progressive Inference and Cross-Transformer1 Nov 2019 0 repositories listed
-
Granular Multimodal Attention Networks for Visual Dialog13 Oct 2019 0 repositories listed
-
On Controlled DeEntanglement for Natural Language Processing22 Sep 2019 0 repositories listed
-
11 Sep 2019 0 repositories listed
-
Building Task-Oriented Visual Dialog Systems Through Alternative Optimization Between Dialog Policy and Language Generation6 Sep 2019 0 repositories listed
-
Grounded Agreement Games: Emphasizing Conversational Grounding in Visual Dialogue Settings29 Aug 2019 0 repositories listed
-
Reactive Multi-Stage Feature Fusion for Multimodal Dialogue Modeling14 Aug 2019 0 repositories listed
-
What Should I Ask? Using Conversationally Informative Rewards for Goal-Oriented Visual Dialog28 Jul 2019 0 repositories listed
-
Learning Goal-Oriented Visual Dialog Agents: Imitating and Surpassing Analytic Experts24 Jul 2019 0 repositories listed
Syntology lines on 12 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections