{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vqs-linking-segmentations-to-questions-and","title":"VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation","arxiv_id":"1708.04686","date":"2017-08-15","proceeding":"ICCV 2017 10","authors":["Chuang Gan","Yandong Li","Haoxiang Li","Chen Sun","Boqing Gong"],"abstract":"Rich and dense human labeled datasets are among the main enabling factors for\nthe recent advance on vision-language understanding. Many seemingly distant\nannotations (e.g., semantic segmentation and visual question answering (VQA))\nare inherently connected in that they reveal different levels and perspectives\nof human understandings about the same visual scenes --- and even the same set\nof images (e.g., of COCO). The popularity of COCO correlates those annotations\nand tasks. Explicitly linking them up may significantly benefit both individual\ntasks and the unified vision and language modeling. We present the preliminary\nwork of linking the instance segmentations provided by COCO to the questions\nand answers (QAs) in the VQA dataset, and name the collected links visual\nquestions and segmentation answers (VQS). They transfer human supervision\nbetween the previously separate tasks, offer more effective leverage to\nexisting problems, and also open the door for new research problems and models.\nWe study two applications of the VQS data in this paper: supervised attention\nfor VQA and a novel question-focused semantic segmentation task. For the\nformer, we obtain state-of-the-art results on the VQA real multiple-choice task\nby simply augmenting the multilayer perceptrons with some attention features\nthat are learned using the segmentation-QA links as explicit supervision. To\nput the latter in perspective, we study two plausible methods and compare them\nto an oracle method assuming that the instance segmentations are given at the\ntest stage.","url_abs":"http://arxiv.org/abs/1708.04686v1","url_pdf":"http://arxiv.org/pdf/1708.04686v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vqs-linking-segmentations-to-questions-and","repo_url":"https://github.com/Cold-Winter/vqs","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"caffe2","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1708.04686","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}