{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/co-attending-free-form-regions-and-detections","title":"Co-attending Free-form Regions and Detections with Multi-modal Multiplicative Feature Embedding for Visual Question Answering","arxiv_id":"1711.06794","date":"2017-11-18","proceeding":null,"authors":["Pan Lu","Hongsheng Li","Wei zhang","Jianyong Wang","Xiaogang Wang"],"abstract":"Recently, the Visual Question Answering (VQA) task has gained increasing\nattention in artificial intelligence. Existing VQA methods mainly adopt the\nvisual attention mechanism to associate the input question with corresponding\nimage regions for effective question answering. The free-form region based and\nthe detection-based visual attention mechanisms are mostly investigated, with\nthe former ones attending free-form image regions and the latter ones attending\npre-specified detection-box regions. We argue that the two attention mechanisms\nare able to provide complementary information and should be effectively\nintegrated to better solve the VQA problem. In this paper, we propose a novel\ndeep neural network for VQA that integrates both attention mechanisms. Our\nproposed framework effectively fuses features from free-form image regions,\ndetection boxes, and question representations via a multi-modal multiplicative\nfeature embedding scheme to jointly attend question-related free-form image\nregions and detection boxes for more accurate question answering. The proposed\nmethod is extensively evaluated on two publicly available datasets, COCO-QA and\nVQA, and outperforms state-of-the-art approaches. Source code is available at\nhttps://github.com/lupantech/dual-mfa-vqa.","url_abs":"http://arxiv.org/abs/1711.06794v2","url_pdf":"http://arxiv.org/pdf/1711.06794v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"co-attending-free-form-regions-and-detections","repo_url":"https://github.com/lupantech/dual-mfa-vqa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"torch","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"form","task_name":"Form"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-question-answering-on-coco-visual-1","task":"Visual Question Answering (VQA)","dataset":"COCO Visual Question Answering (VQA) real images 1.0 multiple choice","model":"Dual-MFA","rank_in_archive_order":2,"of":10,"metrics":{"Percentage correct":"70.04"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-coco-visual-4","task":"Visual Question Answering (VQA)","dataset":"COCO Visual Question Answering (VQA) real images 1.0 open ended","model":"Dual-MFA","rank_in_archive_order":2,"of":14,"metrics":{"Percentage correct":"66.09"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1711.06794","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}