{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/video-transformer-network","title":"Video Transformer Network","arxiv_id":"2102.00719","date":"2021-02-01","proceeding":null,"authors":["Daniel Neimark","Omri Bar","Maya Zohar","Dotan Asselmann"],"abstract":"This paper presents VTN, a transformer-based framework for video recognition. Inspired by recent developments in vision transformers, we ditch the standard approach in video action recognition that relies on 3D ConvNets and introduce a method that classifies actions by attending to the entire video sequence information. Our approach is generic and builds on top of any given 2D spatial network. In terms of wall runtime, it trains $16.1\\times$ faster and runs $5.1\\times$ faster during inference while maintaining competitive accuracy compared to other state-of-the-art methods. It enables whole video analysis, via a single end-to-end pass, while requiring $1.5\\times$ fewer GFLOPs. We report competitive results on Kinetics-400 and present an ablation study of VTN properties and the trade-off between accuracy and inference speed. We hope our approach will serve as a new baseline and start a fresh line of research in the video recognition domain. Code and models are available at: https://github.com/bomri/SlowFast/blob/master/projects/vtn/README.md","url_abs":"https://arxiv.org/abs/2102.00719v3","url_pdf":"https://arxiv.org/pdf/2102.00719v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"video-transformer-network","repo_url":"https://github.com/bomri/SlowFast/blob/master/projects/vtn/README.md","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"action-classification","task_name":"Action Classification"},{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"action-recognition-in-videos-2","task_name":"Action Recognition In Videos"},{"task_slug":"video-recognition","task_name":"Video Recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"ViT-B-VTN+ ImageNet-21K (84.0 [10])","rank_in_archive_order":107,"of":207,"metrics":{"Acc@1":"79.8"},"uses_additional_data":false},{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"ViT-B-VTN (3 layers, ImageNet pretrain)","rank_in_archive_order":125,"of":207,"metrics":{"Acc@1":"78.6","Acc@5":"93.7"},"uses_additional_data":false},{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"ViT-B-VTN+ ImageNet-21K (84.0 [10])","rank_in_archive_order":204,"of":207,"metrics":{"Acc@5":"94.2"},"uses_additional_data":false},{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"ViT-B-VTN (1 layer, ImageNet pretrain)","rank_in_archive_order":206,"of":207,"metrics":{"Acc@5":"93.4"},"uses_additional_data":false},{"leaderboard":"/sota/action-classification-on-moments-in-time","task":"Action Classification","dataset":"MiT","model":"VTN","rank_in_archive_order":12,"of":29,"metrics":{"Top 1 Accuracy":"37.4","Top 5 Accuracy":"65.4"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2102.00719","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}