{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/decoupled-box-proposal-and-featurization-with","title":"Decoupled Box Proposal and Featurization with Ultrafine-Grained Semantic Labels Improve Image Captioning and Visual Question Answering","arxiv_id":"1909.02097","date":"2019-09-04","proceeding":"IJCNLP 2019 11","authors":["Soravit Changpinyo","Bo Pang","Piyush Sharma","Radu Soricut"],"abstract":"Object detection plays an important role in current solutions to vision and language tasks like image captioning and visual question answering. However, popular models like Faster R-CNN rely on a costly process of annotating ground-truths for both the bounding boxes and their corresponding semantic labels, making it less amenable as a primitive task for transfer learning. In this paper, we examine the effect of decoupling box proposal and featurization for down-stream tasks. The key insight is that this allows us to leverage a large amount of labeled annotations that were previously unavailable for standard object detection benchmarks. Empirically, we demonstrate that this leads to effective transfer learning and improved image captioning and visual question answering models, as measured on publicly available benchmarks.","url_abs":"https://arxiv.org/abs/1909.02097v1","url_pdf":"https://arxiv.org/pdf/1909.02097v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"object","task_name":"Object"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"faster-r-cnn","method_name":"Faster R-CNN"},{"method_slug":"rpn","method_name":"RPN"},{"method_slug":"roipool","method_name":"RoIPool"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-question-answering-on-vizwiz-2018-1","task":"Visual Question Answering (VQA)","dataset":"VizWiz 2018","model":"B-Ultra","rank_in_archive_order":4,"of":10,"metrics":{"number":"28.81","other":"35.41","overall":"53.68","unanswerable":"84.03","yes/no":"68.12"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/1909.02097","atlas_url":"https://app.syntology.ai/?focus=1909.02097","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}