Mogrifier LSTM

The Mogrifier LSTM is an extension to the LSTM where the LSTM’s input 𝐱 is gated conditioned on the output of the previous step 𝐡ₚᵣₑᵥ. Next, the gated input is used in a similar manner to gate the output of the previous time step. After a couple of rounds of this mutual gating, the last updated 𝐱 and 𝐡ₚᵣₑᵥ are fed to an LSTM.

In detail, the Mogrifier is an LSTM where two inputs 𝐱 and 𝐡ₚᵣₑᵥ modulate one another in an alternating fashion before the usual LSTM computation takes place. That is: Mogrify(𝐱, 𝐜ₚᵣₑᵥ, 𝐡ₚᵣₑᵥ) = LSTM(𝐱^↑, 𝐜ₚᵣₑᵥ, 𝐡^↑ₚᵣₑᵥ) where the modulated inputs 𝐱^↑ and 𝐡^↑ₚᵣₑᵥ are defined as the highest indexed 𝐱ⁱ and 𝐡ⁱₚᵣₑᵥ, respectively, from the interleaved sequences:

𝐱ⁱ = 2σ(𝐐ⁱ𝐡ⁱ⁻¹ₚᵣₑᵥ) ⊙xⁱ⁻² for odd i ∈[1 …r]

𝐡ⁱₚᵣₑᵥ = 2σ(𝐑ⁱ𝐱ⁱ⁻¹) ⊙𝐡ⁱ⁻²ₚᵣₑᵥ for even i ∈[1 …r]

with 𝐱⁻¹ = 𝐱 and 𝐡⁰ₚᵣₑᵥ = 𝐡ₚᵣₑᵥ. The number of "rounds", r ∈ℕ, is a hyperparameter; r = 0 recovers the LSTM. Multiplication with the constant 2 ensures that randomly initialized 𝐐ⁱ, 𝐑ⁱ matrices result in transformations close to identity. To reduce the number of additional model parameters, we typically factorize the 𝐐ⁱ, 𝐑ⁱ matrices as products of low-rank matrices: 𝐐ⁱ = 𝐐ⁱ_(left)𝐐ⁱ_(right) with 𝐐ⁱ ∈ℝ^(m×n), 𝐐ⁱ_(left) ∈ℝ^(m×k), 𝐐ⁱ_(right) ∈ℝ^(k×n), where k < min(m, n) is the rank.