{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tag-boosting-text-vqa-via-text-aware-visual","title":"TAG: Boosting Text-VQA via Text-aware Visual Question-answer Generation","arxiv_id":"2208.01813","date":"2022-08-03","proceeding":null,"authors":["Jun Wang","Mingfei Gao","Yuqian Hu","Ramprasaath R. Selvaraju","Chetan Ramaiah","ran Xu","Joseph F. JaJa","Larry S. Davis"],"abstract":"Text-VQA aims at answering questions that require understanding the textual cues in an image. Despite the great progress of existing Text-VQA methods, their performance suffers from insufficient human-labeled question-answer (QA) pairs. However, we observe that, in general, the scene text is not fully exploited in the existing datasets -- only a small portion of the text in each image participates in the annotated QA activities. This results in a huge waste of useful information. To address this deficiency, we develop a new method to generate high-quality and diverse QA pairs by explicitly utilizing the existing rich text available in the scene context of each image. Specifically, we propose, TAG, a text-aware visual question-answer generation architecture that learns to produce meaningful, and accurate QA samples using a multimodal transformer. The architecture exploits underexplored scene text information and enhances scene understanding of Text-VQA models by combining the generated QA pairs with the initial training data. Extensive experimental results on two well-known Text-VQA benchmarks (TextVQA and ST-VQA) demonstrate that our proposed TAG effectively enlarges the training data that helps improve the Text-VQA performance without extra labeling effort. Moreover, our model outperforms state-of-the-art approaches that are pre-trained with extra large-scale data. Code is available at https://github.com/HenryJunW/TAG.","url_abs":"https://arxiv.org/abs/2208.01813v3","url_pdf":"https://arxiv.org/pdf/2208.01813v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tag-boosting-text-vqa-via-text-aware-visual","repo_url":"https://github.com/HenryJunW/TAG","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"answer-generation","task_name":"Answer Generation"},{"task_slug":"question-answer-generation","task_name":"Question-Answer-Generation"},{"task_slug":"scene-understanding","task_name":"Scene Understanding"},{"task_slug":"tag","task_name":"TAG"},{"task_slug":null,"task_name":"TextVQA"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-question-answering-on-textvqa-test-1","task":"Visual Question Answering (VQA)","dataset":"TextVQA test-standard","model":"TAG","rank_in_archive_order":3,"of":12,"metrics":{"overall":"53.69"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2208.01813","atlas_url":"https://app.syntology.ai/?focus=2208.01813","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}