{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ask-attend-and-answer-exploring-question","title":"Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering","arxiv_id":"1511.05234","date":"2015-11-17","proceeding":null,"authors":["Huijuan Xu","Kate Saenko"],"abstract":"We address the problem of Visual Question Answering (VQA), which requires\njoint image and language understanding to answer a question about a given\nphotograph. Recent approaches have applied deep image captioning methods based\non convolutional-recurrent networks to this problem, but have failed to model\nspatial inference. To remedy this, we propose a model we call the Spatial\nMemory Network and apply it to the VQA task. Memory networks are recurrent\nneural networks with an explicit attention mechanism that selects certain parts\nof the information stored in memory. Our Spatial Memory Network stores neuron\nactivations from different spatial regions of the image in its memory, and uses\nthe question to choose relevant regions for computing the answer, a process of\nwhich constitutes a single \"hop\" in the network. We propose a novel spatial\nattention architecture that aligns words with image patches in the first hop,\nand obtain improved results by adding a second attention hop which considers\nthe whole question to choose visual evidence based on the results of the first\nhop. To better understand the inference process learned by the network, we\ndesign synthetic questions that specifically require spatial inference and\nvisualize the attention weights. We evaluate our model on two published visual\nquestion answering datasets, DAQUAR [1] and VQA [2], and obtain improved\nresults compared to a strong deep baseline model (iBOWIMG) which concatenates\nimage and question features to predict the answer [3].","url_abs":"http://arxiv.org/abs/1511.05234v2","url_pdf":"http://arxiv.org/pdf/1511.05234v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ask-attend-and-answer-exploring-question","repo_url":"https://github.com/zixuwang1996/VQA-reading-list","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[{"method_slug":"memory-network","method_name":"Memory Network"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-question-answering-on-coco-visual-4","task":"Visual Question Answering (VQA)","dataset":"COCO Visual Question Answering (VQA) real images 1.0 open ended","model":"SMem-VQA","rank_in_archive_order":13,"of":14,"metrics":{"Percentage correct":"58.2"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1511.05234","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}