Spatial and Channel-wise Attention-based Convolutional Neural Network

As CNN features are naturally spatial, channel-wise and multi-layer, Chen et al. proposed a novel spatial and channel-wise attention-based convolutional neural network (SCA-CNN). It was designed for the task of image captioning, and uses an encoder-decoder framework where a CNN first encodes an input image into a vector and then an LSTM decodes the vector into a sequence of words. Given an input feature map X and the previous time step LSTM hidden state hₜ₋₁ ∈ℝᵈ, a spatial attention mechanism pays more attention to the semantically useful regions, guided by LSTM hidden state hₜ₋₁. The spatial attention model is:

a(hₜ₋₁, X) = tanh(Conv₁^(1 ×1)(X) ⊕W₁ hₜ₋₁)

Φₛ(hₜ₋₁, X) = Softmax(Conv₂^(1 ×1)(a(hₜ₋₁, X)))

where ⊕ represents addition of a matrix and a vector. Similarly, channel-wise attention aggregates global information first, and then computes a channel-wise attention weight vector with the hidden state hₜ₋₁: b(hₜ₋₁, X) = tanh((W₂GAP(X)+b₂)⊕W₁hₜ₋₁) Φ_c(hₜ₋₁, X) = Softmax(W₃(b(hₜ₋₁, X))+b₃) Overall, the SCA mechanism can be written in one of two ways. If channel-wise attention is applied before spatial attention, we have Y = f(X,Φₛ(hₜ₋₁, X Φ_c(hₜ₋₁, X)), Φ_c(hₜ₋₁, X)) and if spatial attention comes first: Y = f(X,Φₛ(hₜ₋₁, X), Φ_c(hₜ₋₁, X Φₛ(hₜ₋₁, X))) where f(·) denotes the modulate function which takes the feature map X and attention maps as input and then outputs the modulated feature map Y.

Unlike previous attention mechanisms which consider each image region equally and use global spatial information to tell the network where to focus, SCA-Net leverages the semantic vector to produce the spatial attention map as well as the channel-wise attention weight vector. Being more than a powerful attention model, SCA-CNN also provides a better understanding of where and what the model should focus on during sentence generation.