Dense Synthesized Attention

Dense Synthesized Attention, introduced with the Synthesizer architecture, is a type of synthetic attention mechanism that replaces the notion of query-key-values in the self-attention module and directly synthesizes the alignment matrix instead. Dense attention is conditioned on each input token. The method accepts an input X ∈ℝ^(l x d) and produces an output of Y ∈ℝ^(l x d). Here l refers to the sequence length and d refers to the dimensionality of the model. We first adopt F(.), a parameterized function, for projecting input Xᵢ from d dimensions to l dimensions.

Bᵢ = F(Xᵢ)

where F(.) is a parameterized function that maps ℝᵈ to ℝˡ and i is the i-th token of X. Intuitively, this can be interpreted as learning a token-wise projection to the sequence length l. Essentially, with this model, each token predicts weights for each token in the input sequence. In practice, a simple two layered feed-forward layer with ReLU activations for F(.) is adopted:

F(X) = W(σ_R(W(X) + b)) + b

where σ_R is the ReLU activation function. Hence, B is now of ℝ^(l x d). Given B, we now compute:

Y = Softmax(B)G(X)

where G(.) is another parameterized function of X that is analogous to V (value) in the standard Transformer model. This approach eliminates the dot product altogether by replacing QKᵀ in standard Transformers with the synthesizing function F(.).