{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/video-graph-transformer-for-video-question","title":"Video Graph Transformer for Video Question Answering","arxiv_id":"2207.05342","date":"2022-07-12","proceeding":null,"authors":["Junbin Xiao","Pan Zhou","Tat-Seng Chua","Shuicheng Yan"],"abstract":"This paper proposes a Video Graph Transformer (VGT) model for Video Quetion Answering (VideoQA). VGT's uniqueness are two-fold: 1) it designs a dynamic graph transformer module which encodes video by explicitly capturing the visual objects, their relations, and dynamics for complex spatio-temporal reasoning; and 2) it exploits disentangled video and text Transformers for relevance comparison between the video and text to perform QA, instead of entangled cross-modal Transformer for answer classification. Vision-text communication is done by additional cross-modal interaction modules. With more reasonable video encoding and QA solution, we show that VGT can achieve much better performances on VideoQA tasks that challenge dynamic relation reasoning than prior arts in the pretraining-free scenario. Its performances even surpass those models that are pretrained with millions of external data. We further show that VGT can also benefit a lot from self-supervised cross-modal pretraining, yet with orders of magnitude smaller data. These results clearly demonstrate the effectiveness and superiority of VGT, and reveal its potential for more data-efficient pretraining. With comprehensive analyses and some heuristic observations, we hope that VGT can promote VQA research beyond coarse recognition/description towards fine-grained relation reasoning in realistic videos. Our code is available at https://github.com/sail-sg/VGT.","url_abs":"https://arxiv.org/abs/2207.05342v3","url_pdf":"https://arxiv.org/pdf/2207.05342v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"video-graph-transformer-for-video-question","repo_url":"https://github.com/sail-sg/vgt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":null,"task_name":"Relation"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"graph-transformer","method_name":"Graph Transformer"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"lapeigen","method_name":"LapEigen"},{"method_slug":"laplacian-pe","method_name":"Laplacian PE"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-question-answering-on-intentqa","task":"Video Question Answering","dataset":"IntentQA","model":"VGT","rank_in_archive_order":5,"of":6,"metrics":{"Accuarcy":"51.3","CH":"56.0","CW":"51.4","TP&TN":"47.6"},"uses_additional_data":false},{"leaderboard":"/sota/video-question-answering-on-next-qa","task":"Video Question Answering","dataset":"NExT-QA","model":"VGT(PT)","rank_in_archive_order":42,"of":47,"metrics":{"Accuracy":"56.9"},"uses_additional_data":true},{"leaderboard":"/sota/video-question-answering-on-next-qa","task":"Video Question Answering","dataset":"NExT-QA","model":"VGT","rank_in_archive_order":44,"of":47,"metrics":{"Accuracy":"55.0"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2207.05342","atlas_url":"https://app.syntology.ai/?focus=2207.05342","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2207.05342"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sail-sg/vgt","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/sail-sg/VGT","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":8,"ran_draft_wrong":1,"unverified":5},"by_repo_kind":{"official":{"samples":14,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f3ff51937dd98c21","entry":"CMAtten","repo":"sail-sg/VGT","repo_kind":"official","path":"model/vqa_model.py","file_url":"https://github.com/sail-sg/VGT/blob/HEAD/model/vqa_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f3ff51937dd98c21"}},{"code_sha256_prefix":"5a6a727d5b0f9c6b","entry":"Embeddings","repo":"sail-sg/VGT","repo_kind":"official","path":"model/vqa_model.py","file_url":"https://github.com/sail-sg/VGT/blob/HEAD/model/vqa_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5a6a727d5b0f9c6b"}},{"code_sha256_prefix":"66dcb62d12860979","entry":"EncoderVid","repo":"sail-sg/VGT","repo_kind":"official","path":"model/vqa_model.py","file_url":"https://github.com/sail-sg/VGT/blob/HEAD/model/vqa_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"66dcb62d12860979"}},{"code_sha256_prefix":"fad49f16c43d52df","entry":"FFN","repo":"sail-sg/VGT","repo_kind":"official","path":"model/vqa_model.py","file_url":"https://github.com/sail-sg/VGT/blob/HEAD/model/vqa_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fad49f16c43d52df"}},{"code_sha256_prefix":"c7325ef7876ef60e","entry":"Graph","repo":"sail-sg/VGT","repo_kind":"official","path":"model/vqa_model.py","file_url":"https://github.com/sail-sg/VGT/blob/HEAD/model/vqa_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c7325ef7876ef60e"}},{"code_sha256_prefix":"889a03d6a437ba1c","entry":"GraphConvolution","repo":"sail-sg/VGT","repo_kind":"official","path":"model/vqa_model.py","file_url":"https://github.com/sail-sg/VGT/blob/HEAD/model/vqa_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"889a03d6a437ba1c"}},{"code_sha256_prefix":"b091b2a22b3049e3","entry":"MultiHeadSelfAttention","repo":"sail-sg/VGT","repo_kind":"official","path":"model/vqa_model.py","file_url":"https://github.com/sail-sg/VGT/blob/HEAD/model/vqa_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b091b2a22b3049e3"}},{"code_sha256_prefix":"b294b4902336fb78","entry":"TransformerBlock","repo":"sail-sg/VGT","repo_kind":"official","path":"model/vqa_model.py","file_url":"https://github.com/sail-sg/VGT/blob/HEAD/model/vqa_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b294b4902336fb78"}},{"code_sha256_prefix":"4b0473dade9942bc","entry":"get_mask","repo":"sail-sg/VGT","repo_kind":"official","path":"model/vqa_model.py","file_url":"https://github.com/sail-sg/VGT/blob/HEAD/model/vqa_model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4b0473dade9942bc"}},{"code_sha256_prefix":"2f0df7b215389abb","entry":"AModel","repo":"sail-sg/VGT","repo_kind":"official","path":"model/vqa_model.py","file_url":"https://github.com/sail-sg/VGT/blob/HEAD/model/vqa_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2f0df7b215389abb"}},{"code_sha256_prefix":"f64528547183d480","entry":"Bert","repo":"sail-sg/VGT","repo_kind":"official","path":"model/vqa_model.py","file_url":"https://github.com/sail-sg/VGT/blob/HEAD/model/vqa_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f64528547183d480"}},{"code_sha256_prefix":"9d80cb2ca7b058a1","entry":"Transformer","repo":"sail-sg/VGT","repo_kind":"official","path":"model/vqa_model.py","file_url":"https://github.com/sail-sg/VGT/blob/HEAD/model/vqa_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9d80cb2ca7b058a1"}},{"code_sha256_prefix":"cdedc918d8a5dc8b","entry":"VGT","repo":"sail-sg/VGT","repo_kind":"official","path":"model/vqa_model.py","file_url":"https://github.com/sail-sg/VGT/blob/HEAD/model/vqa_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cdedc918d8a5dc8b"}},{"code_sha256_prefix":"741ba8df50c491fc","entry":"create_sinusoidal_embeddings","repo":"sail-sg/VGT","repo_kind":"official","path":"model/vqa_model.py","file_url":"https://github.com/sail-sg/VGT/blob/HEAD/model/vqa_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"741ba8df50c491fc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}