Spatio-Temporal Attention LSTM

In human action recognition, each type of action generally only depends on a few specific kinematic joints. Furthermore, over time, multiple actions may be performed. Motivated by these observations, Song et al. proposed a joint spatial and temporal attention network based on LSTM, to adaptively find discriminative features and keyframes. Its main attention-related components are a spatial attention sub-network, to select important regions, and a temporal attention sub-network, to select key frames. The spatial attention sub-network can be written as: sₜ = Uₛtanh(WₓₛXₜ + Wₕₛhₜ₋₁ˢ + bₛᵢ) + bₛₒ αₜ = Softmax(sₜ) Yₜ = αₜ Xₜ where Xₜ is the input feature at time t, Uₛ, Wₕₛ, bₛᵢ, and bₛₒ are learnable parameters, and hₜ₋₁ˢ is the hidden state at step t-1. Note that use of the hidden state h means the attention process takes temporal relationships into consideration.

The temporal attention sub-network is similar to the spatial branch and produces its attention map using: βₜ = δ(WₓₚXₜ + Wₕₚhₜ₋₁ᵖ + bₚ). It adopts a ReLU function instead of a normalization function for ease of optimization. It also uses a regularized objective function to improve convergence.

Overall, this paper presents a joint spatiotemporal attention method to focus on important joints and keyframes, with excellent results on the action recognition task.