{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/an-image-is-worth-16x16-words-what-is-a-video","title":"An Image is Worth 16x16 Words, What is a Video Worth?","arxiv_id":"2103.13915","date":"2021-03-25","proceeding":null,"authors":["Gilad Sharir","Asaf Noy","Lihi Zelnik-Manor"],"abstract":"Leading methods in the domain of action recognition try to distill information from both the spatial and temporal dimensions of an input video. Methods that reach State of the Art (SotA) accuracy, usually make use of 3D convolution layers as a way to abstract the temporal information from video frames. The use of such convolutions requires sampling short clips from the input video, where each clip is a collection of closely sampled frames. Since each short clip covers a small fraction of an input video, multiple clips are sampled at inference in order to cover the whole temporal length of the video. This leads to increased computational load and is impractical for real-world applications. We address the computational bottleneck by significantly reducing the number of frames required for inference. Our approach relies on a temporal transformer that applies global attention over video frames, and thus better exploits the salient information in each frame. Therefore our approach is very input efficient, and can achieve SotA results (on Kinetics dataset) with a fraction of the data (frames per video), computation and latency. Specifically on Kinetics-400, we reach $80.5$ top-1 accuracy with $\\times 30$ less frames per video, and $\\times 40$ faster inference than the current leading method. Code is available at: https://github.com/Alibaba-MIIL/STAM","url_abs":"https://arxiv.org/abs/2103.13915v2","url_pdf":"https://arxiv.org/pdf/2103.13915v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"an-image-is-worth-16x16-words-what-is-a-video","repo_url":"https://github.com/Alibaba-MIIL/STAM","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"an-image-is-worth-16x16-words-what-is-a-video","repo_url":"https://github.com/lucidrains/STAM-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"action-classification","task_name":"Action Classification"},{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"}],"methods":[{"method_slug":"3d-convolution","method_name":"3D Convolution"},{"method_slug":"convolution","method_name":"Convolution"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"STAM (64 Frames)","rank_in_archive_order":99,"of":207,"metrics":{"Acc@1":"80.5","FLOPs (G) x views":"1040x1"},"uses_additional_data":false},{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"STAM (16 Frames)","rank_in_archive_order":114,"of":207,"metrics":{"Acc@1":"79.3","FLOPs (G) x views":"270x1"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-ucf101","task":"Action Recognition","dataset":"UCF101","model":"STAM-32 (ImageNet/Kinetics pretraining)","rank_in_archive_order":28,"of":91,"metrics":{"3-fold Accuracy":"97"},"uses_additional_data":true}],"syntology":{"syntology_url":"https://syntology.ai/paper/2103.13915","atlas_url":"https://app.syntology.ai/?focus=2103.13915","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2103.13915"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lucidrains/STAM-pytorch","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Alibaba-MIIL/STAM","reach":null}],"summary":{"unverified":1},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"49e68764c5d58f0a","entry":"STAM_224","repo":"Alibaba-MIIL/STAM","repo_kind":"official","path":"stam/models/transformer_model.py","file_url":"https://github.com/Alibaba-MIIL/STAM/blob/HEAD/stam/models/transformer_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"49e68764c5d58f0a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}