Papers › Knowing What, Where and When to Look: Efficient Video Action Modeling with Attention
Knowing What, Where and When to Look: Efficient Video Action Modeling with Attention
Juan-Manuel Perez-Rua, Brais Martinez, Xiatian Zhu, Antoine Toisoul, Victor Escorcia, Tao Xiang
Attentive video modeling is essential for action recognition in unconstrained videos due to their rich yet redundant information over space and time. However, introducing attention in a deep neural network for action recognition is challenging for two reasons. First, an effective attention module needs to learn what (objects and their local motion patterns), where (spatially), and when (temporally) to focus on. Second, a video attention module must be efficient because existing action recognition models already suffer from high computational cost. To address both challenges, a novel What-Where-When (W3) video attention module is proposed. Departing from existing alternatives, our W3 module models all three facets of video attention jointly. Crucially, it is extremely efficient by factorizing the high-dimensional video feature data into low-dimensional meaningful spaces (1D channel vector for `what' and 2D spatial tensors for `where'), followed by lightweight temporal attention reasoning. Extensive experiments show that our attention model brings significant improvements to existing action recognition models, achieving new state-of-the-art performance on a number of benchmarks.
Code
No code repository is listed for this paper in the archive or in Syntology's graph.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Action Recognition | EPIC-KITCHENS-55 | TSM+W3 - full res | Top-1 Accuracy | 34.2 | #1 of 1 | Archive leaderboard | report |
| Action Recognition | EgoGesture | TSM+W3 | Top-1 Accuracy | 94.3 | #1 of 1 | Archive leaderboard | report |
| Action Recognition | EgoGesture | TSM+W3 | Top-5 Accuracy | 99.2 | #1 of 1 | Archive leaderboard | report |
| Action Recognition | Something-Something V1 | TSM+W3 (16 frames, ResNet50) | Top 1 Accuracy | 52.6 | #42 of 74 | Archive leaderboard | report |
| Action Recognition | Something-Something V1 | TSM+W3 (16 frames, ResNet50) | Top 5 Accuracy | 81.3 | #42 of 74 | Archive leaderboard | report |
| Action Recognition | Something-Something V2 | TSM+W3 (16 frames, RGB ResNet-50) | Top-1 Accuracy | 66.5 | #79 of 123 | Archive leaderboard | report |
| Action Recognition | Something-Something V2 | TSM+W3 (16 frames, RGB ResNet-50) | Top-5 Accuracy | 90.4 | #79 of 123 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections