{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tgif-qa-toward-spatio-temporal-reasoning-in","title":"TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering","arxiv_id":"1704.04497","date":"2017-04-14","proceeding":"CVPR 2017 7","authors":["Yunseok Jang","Yale Song","Youngjae Yu","Youngjin Kim","Gunhee Kim"],"abstract":"Vision and language understanding has emerged as a subject undergoing intense\nstudy in Artificial Intelligence. Among many tasks in this line of research,\nvisual question answering (VQA) has been one of the most successful ones, where\nthe goal is to learn a model that understands visual content at region-level\ndetails and finds their associations with pairs of questions and answers in the\nnatural language form. Despite the rapid progress in the past few years, most\nexisting work in VQA have focused primarily on images. In this paper, we focus\non extending VQA to the video domain and contribute to the literature in three\nimportant ways. First, we propose three new tasks designed specifically for\nvideo VQA, which require spatio-temporal reasoning from videos to answer\nquestions correctly. Next, we introduce a new large-scale dataset for video VQA\nnamed TGIF-QA that extends existing VQA work with our new tasks. Finally, we\npropose a dual-LSTM based approach with both spatial and temporal attention,\nand show its effectiveness over conventional VQA techniques through empirical\nevaluations.","url_abs":"http://arxiv.org/abs/1704.04497v3","url_pdf":"http://arxiv.org/pdf/1704.04497v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tgif-qa-toward-spatio-temporal-reasoning-in","repo_url":"https://github.com/ahjeongseo/MASN-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"tgif-qa-toward-spatio-temporal-reasoning-in","repo_url":"https://github.com/chaitanyadwivedii/3D-Attention-is-All-You-Need","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"zeroshot-video-question-answer","task_name":"Zero-Shot Video Question Answer"}],"methods":[],"datasets_introduced":[{"slug":"tgif-qa","name":"TGIF-QA","full_name":"TGIF-QA"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/visual-question-answering-on-msrvtt-qa-1","task":"Visual Question Answering (VQA)","dataset":"MSRVTT-QA","model":"ST-VQA","rank_in_archive_order":33,"of":34,"metrics":{"Accuracy":"0.309"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-msvd-qa-1","task":"Visual Question Answering (VQA)","dataset":"MSVD-QA","model":"ST-VQA","rank_in_archive_order":36,"of":36,"metrics":{"Accuracy":"0.313"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/1704.04497","atlas_url":"https://app.syntology.ai/?focus=1704.04497","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}