{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/r-vqa-learning-visual-relation-facts-with","title":"R-VQA: Learning Visual Relation Facts with Semantic Attention for Visual Question Answering","arxiv_id":"1805.09701","date":"2018-05-24","proceeding":null,"authors":["Pan Lu","Lei Ji","Wei zhang","Nan Duan","Ming Zhou","Jianyong Wang"],"abstract":"Recently, Visual Question Answering (VQA) has emerged as one of the most\nsignificant tasks in multimodal learning as it requires understanding both\nvisual and textual modalities. Existing methods mainly rely on extracting image\nand question features to learn their joint feature embedding via multimodal\nfusion or attention mechanism. Some recent studies utilize external\nVQA-independent models to detect candidate entities or attributes in images,\nwhich serve as semantic knowledge complementary to the VQA task. However, these\ncandidate entities or attributes might be unrelated to the VQA task and have\nlimited semantic capacities. To better utilize semantic knowledge in images, we\npropose a novel framework to learn visual relation facts for VQA. Specifically,\nwe build up a Relation-VQA (R-VQA) dataset based on the Visual Genome dataset\nvia a semantic similarity module, in which each data consists of an image, a\ncorresponding question, a correct answer and a supporting relation fact. A\nwell-defined relation detector is then adopted to predict visual\nquestion-related relation facts. We further propose a multi-step attention\nmodel composed of visual attention and semantic attention sequentially to\nextract related visual knowledge and semantic knowledge. We conduct\ncomprehensive experiments on the two benchmark datasets, demonstrating that our\nmodel achieves state-of-the-art performance and verifying the benefit of\nconsidering visual relation facts.","url_abs":"http://arxiv.org/abs/1805.09701v2","url_pdf":"http://arxiv.org/pdf/1805.09701v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"r-vqa-learning-visual-relation-facts-with","repo_url":"https://github.com/lupantech/rvqa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":null,"task_name":"Relation"},{"task_slug":"semantic-similarity","task_name":"Semantic Similarity"},{"task_slug":"semantic-textual-similarity","task_name":"Semantic Textual Similarity"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-question-answering-on-coco-visual-1","task":"Visual Question Answering (VQA)","dataset":"COCO Visual Question Answering (VQA) real images 1.0 multiple choice","model":"RelAtt","rank_in_archive_order":3,"of":10,"metrics":{"Percentage correct":"69.60"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-coco-visual-4","task":"Visual Question Answering (VQA)","dataset":"COCO Visual Question Answering (VQA) real images 1.0 open ended","model":"RelAtt","rank_in_archive_order":4,"of":14,"metrics":{"Percentage correct":"65.69"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1805.09701","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}