{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/visual-dialog","title":"Visual Dialog","arxiv_id":"1611.08669","date":"2016-11-26","proceeding":"CVPR 2017 7","authors":["Abhishek Das","Satwik Kottur","Khushi Gupta","Avi Singh","Deshraj Yadav","José M. F. Moura","Devi Parikh","Dhruv Batra"],"abstract":"We introduce the task of Visual Dialog, which requires an AI agent to hold a\nmeaningful dialog with humans in natural, conversational language about visual\ncontent. Specifically, given an image, a dialog history, and a question about\nthe image, the agent has to ground the question in image, infer context from\nhistory, and answer the question accurately. Visual Dialog is disentangled\nenough from a specific downstream task so as to serve as a general test of\nmachine intelligence, while being grounded in vision enough to allow objective\nevaluation of individual responses and benchmark progress. We develop a novel\ntwo-person chat data-collection protocol to curate a large-scale Visual Dialog\ndataset (VisDial). VisDial v0.9 has been released and contains 1 dialog with 10\nquestion-answer pairs on ~120k images from COCO, with a total of ~1.2M dialog\nquestion-answer pairs.\n  We introduce a family of neural encoder-decoder models for Visual Dialog with\n3 encoders -- Late Fusion, Hierarchical Recurrent Encoder and Memory Network --\nand 2 decoders (generative and discriminative), which outperform a number of\nsophisticated baselines. We propose a retrieval-based evaluation protocol for\nVisual Dialog where the AI agent is asked to sort a set of candidate answers\nand evaluated on metrics such as mean-reciprocal-rank of human response. We\nquantify gap between machine and human performance on the Visual Dialog task\nvia human studies. Putting it all together, we demonstrate the first 'visual\nchatbot'! Our dataset, code, trained models and visual chatbot are available on\nhttps://visualdialog.org","url_abs":"http://arxiv.org/abs/1611.08669v5","url_pdf":"http://arxiv.org/pdf/1611.08669v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"visual-dialog","repo_url":"https://github.com/batra-mlp-lab/visdial-amt-chat","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"visual-dialog","repo_url":"https://github.com/Cloud-CV/visual-chatbot","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"visual-dialog","repo_url":"https://github.com/batra-mlp-lab/visdial","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"torch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"visual-dialog","repo_url":"https://github.com/batra-mlp-lab/visdial-challenge-starter-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"visual-dialog","repo_url":"https://github.com/facebookresearch/ParlAI","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"visual-dialog","repo_url":"https://github.com/gicheonkang/gst-visdial","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"visual-dialog","repo_url":"https://github.com/joe-prog/https-github.com-facebookresearch-ParlAI","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"gone","observed_at":"2026-09-18","how":"tree_404+repo_404"}},{"paper_slug":"visual-dialog","repo_url":"https://github.com/kdexd/lang-emerge-parlai","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"visual-dialog","repo_url":"https://github.com/nonslowrunner/Visualdialogs","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"visual-dialog","repo_url":"https://github.com/yuleiniu/rva","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"visual-dialog","repo_url":"https://github.com/taesunwhang/MVAN-VisDial","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"ai-agent","task_name":"AI Agent"},{"task_slug":"chatbot","task_name":"Chatbot"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"visual-dialogue","task_name":"Visual Dialog"}],"methods":[{"method_slug":"memory-network","method_name":"Memory Network"}],"datasets_introduced":[{"slug":"visdial","name":"VisDial","full_name":"Visual Dialog"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-dialog-on-visdial-v09-val","task":"Visual Dialog","dataset":"VisDial v0.9 val","model":"MN-QIH-D","rank_in_archive_order":15,"of":18,"metrics":{"MRR":"0.5965","Mean Rank":"5.46","R@1":"45.55","R@10":"85.37","R@5":"76.22"},"uses_additional_data":false},{"leaderboard":"/sota/visual-dialog-on-visdial-v09-val","task":"Visual Dialog","dataset":"VisDial v0.9 val","model":"HRE-QIH-D","rank_in_archive_order":16,"of":18,"metrics":{"MRR":"0.5846","Mean Rank":"5.72","R@1":"44.67","R@10":"84.22","R@5":"74.50"},"uses_additional_data":false},{"leaderboard":"/sota/visual-dialog-on-visdial-v09-val","task":"Visual Dialog","dataset":"VisDial v0.9 val","model":"HRE-QIH-D","rank_in_archive_order":17,"of":18,"metrics":{"MRR":"0.5807","Mean Rank":"5.78","R@1":"43.82","R@10":"84.07","R@5":"74.68"},"uses_additional_data":false},{"leaderboard":"/sota/visual-dialog-on-visual-dialog-v1-0-test-std","task":"Visual Dialog","dataset":"Visual Dialog v1.0 test-std","model":"MN-QIH-D","rank_in_archive_order":75,"of":80,"metrics":{"MRR (x 100)":"55.5","Mean":"5.92","NDCG (x 100)":"47.5","R@1":"40.98","R@10":"83.30","R@5":"72.30"},"uses_additional_data":false},{"leaderboard":"/sota/visual-dialog-on-visual-dialog-v1-0-test-std","task":"Visual Dialog","dataset":"Visual Dialog v1.0 test-std","model":"HRE-QIH-D","rank_in_archive_order":77,"of":80,"metrics":{"MRR (x 100)":"54.2","Mean":"6.41","NDCG (x 100)":"45.5","R@1":"39.93","R@10":"81.50","R@5":"70.45"},"uses_additional_data":false},{"leaderboard":"/sota/visual-dialog-on-visual-dialog-v1-0-test-std","task":"Visual Dialog","dataset":"Visual Dialog v1.0 test-std","model":"MN-QIH-D","rank_in_archive_order":78,"of":80,"metrics":{"MRR (x 100)":"55.4","Mean":"5.95","NDCG (x 100)":"45.3","R@1":"40.95","R@10":"82.83","R@5":"72.45"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1611.08669","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1611.08669"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/ParlAI","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/batra-mlp-lab/visdial-challenge-starter-pytorch","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Cloud-CV/visual-chatbot","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gicheonkang/gst-visdial","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kdexd/lang-emerge-parlai","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/batra-mlp-lab/visdial-amt-chat","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nonslowrunner/Visualdialogs","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/joe-prog/https-github.com-facebookresearch-ParlAI","reach":{"status":"gone","observed_at":"2026-09-18","how":"tree_404+repo_404"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/batra-mlp-lab/visdial","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/taesunwhang/MVAN-VisDial","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yuleiniu/rva","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran_draft_wrong":2},"by_repo_kind":{"listed":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"139efc85bd981f9a","entry":"decode_data","repo":"gicheonkang/gst-visdial","repo_kind":"listed","path":"generate.py","file_url":"https://github.com/gicheonkang/gst-visdial/blob/HEAD/generate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"139efc85bd981f9a"}},{"code_sha256_prefix":"b0b960555948cc0d","entry":"forward","repo":"gicheonkang/gst-visdial","repo_kind":"listed","path":"train_gen.py","file_url":"https://github.com/gicheonkang/gst-visdial/blob/HEAD/train_gen.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b0b960555948cc0d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}