{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/training-recurrent-answering-units-with-joint","title":"Training Recurrent Answering Units with Joint Loss Minimization for VQA","arxiv_id":"1606.03647","date":"2016-06-12","proceeding":null,"authors":["Hyeonwoo Noh","Bohyung Han"],"abstract":"We propose a novel algorithm for visual question answering based on a\nrecurrent deep neural network, where every module in the network corresponds to\na complete answering unit with attention mechanism by itself. The network is\noptimized by minimizing loss aggregated from all the units, which share model\nparameters while receiving different information to compute attention\nprobability. For training, our model attends to a region within image feature\nmap, updates its memory based on the question and attended image feature, and\nanswers the question based on its memory state. This procedure is performed to\ncompute loss in each step. The motivation of this approach is our observation\nthat multi-step inferences are often required to answer questions while each\nproblem may have a unique desirable number of steps, which is difficult to\nidentify in practice. Hence, we always make the first unit in the network solve\nproblems, but allow it to learn the knowledge from the rest of units by\nbackpropagation unless it degrades the model. To implement this idea, we\nearly-stop training each unit as soon as it starts to overfit. Note that, since\nmore complex models tend to overfit on easier questions quickly, the last\nanswering unit in the unfolded recurrent neural network is typically killed\nfirst while the first one remains last. We make a single-step prediction for a\nnew question using the shared model. This strategy works better than the other\noptions within our framework since the selected model is trained effectively\nfrom all units without overfitting. The proposed algorithm outperforms other\nmulti-step attention based approaches using a single step prediction in VQA\ndataset.","url_abs":"http://arxiv.org/abs/1606.03647v2","url_pdf":"http://arxiv.org/pdf/1606.03647v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"bottleneck-residual-block","method_name":"Bottleneck Residual Block"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"kaiming-initialization","method_name":"Kaiming Initialization"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-block","method_name":"Residual Block"},{"method_slug":"residual-connection","method_name":"Residual Connection"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-question-answering-on-coco-visual-1","task":"Visual Question Answering (VQA)","dataset":"COCO Visual Question Answering (VQA) real images 1.0 multiple choice","model":"joint-loss","rank_in_archive_order":5,"of":10,"metrics":{"Percentage correct":"67.3"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-coco-visual-4","task":"Visual Question Answering (VQA)","dataset":"COCO Visual Question Answering (VQA) real images 1.0 open ended","model":"joint-loss","rank_in_archive_order":5,"of":14,"metrics":{"Percentage correct":"63.2"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-vqa-v1-test-dev","task":"Visual Question Answering (VQA)","dataset":"VQA v1 test-dev","model":"RAU (ResNet)","rank_in_archive_order":4,"of":7,"metrics":{"Accuracy":"63.3"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-vqa-v1-test-std","task":"Visual Question Answering (VQA)","dataset":"VQA v1 test-std","model":"RAU (ResNet)","rank_in_archive_order":2,"of":6,"metrics":{"Accuracy":"63.2"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1606.03647","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}