{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/knowing-what-where-and-when-to-look-efficient","title":"Knowing What, Where and When to Look: Efficient Video Action Modeling with Attention","arxiv_id":"2004.01278","date":"2020-04-02","proceeding":null,"authors":["Juan-Manuel Perez-Rua","Brais Martinez","Xiatian Zhu","Antoine Toisoul","Victor Escorcia","Tao Xiang"],"abstract":"Attentive video modeling is essential for action recognition in unconstrained videos due to their rich yet redundant information over space and time. However, introducing attention in a deep neural network for action recognition is challenging for two reasons. First, an effective attention module needs to learn what (objects and their local motion patterns), where (spatially), and when (temporally) to focus on. Second, a video attention module must be efficient because existing action recognition models already suffer from high computational cost. To address both challenges, a novel What-Where-When (W3) video attention module is proposed. Departing from existing alternatives, our W3 module models all three facets of video attention jointly. Crucially, it is extremely efficient by factorizing the high-dimensional video feature data into low-dimensional meaningful spaces (1D channel vector for `what' and 2D spatial tensors for `where'), followed by lightweight temporal attention reasoning. Extensive experiments show that our attention model brings significant improvements to existing action recognition models, achieving new state-of-the-art performance on a number of benchmarks.","url_abs":"https://arxiv.org/abs/2004.01278v1","url_pdf":"https://arxiv.org/pdf/2004.01278v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"bottleneck-residual-block","method_name":"Bottleneck Residual Block"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"kaiming-initialization","method_name":"Kaiming Initialization"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-block","method_name":"Residual Block"},{"method_slug":"residual-connection","method_name":"Residual Connection"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-recognition-in-videos-on-epic-kitchens","task":"Action Recognition","dataset":"EPIC-KITCHENS-55","model":"TSM+W3 - full res","rank_in_archive_order":1,"of":1,"metrics":{"Top-1 Accuracy":"34.2"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-egogesture","task":"Action Recognition","dataset":"EgoGesture","model":"TSM+W3","rank_in_archive_order":1,"of":1,"metrics":{"Top-1 Accuracy":"94.3","Top-5 Accuracy":"99.2"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-something-1","task":"Action Recognition","dataset":"Something-Something V1","model":"TSM+W3 (16 frames, ResNet50)","rank_in_archive_order":42,"of":74,"metrics":{"Top 1 Accuracy":"52.6","Top 5 Accuracy":"81.3"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-something","task":"Action Recognition","dataset":"Something-Something V2","model":"TSM+W3 (16 frames, RGB ResNet-50)","rank_in_archive_order":79,"of":123,"metrics":{"Top-1 Accuracy":"66.5","Top-5 Accuracy":"90.4"},"uses_additional_data":true}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}