Deep LSTM Reader

The Deep LSTM Reader is a neural network for reading comprehension. We feed documents one word at a time into a Deep LSTM encoder, after a delimiter we then also feed the query into the encoder. The model therefore processes each document query pair as a single long sequence. Given the embedded document and query the network predicts which token in the document answers the query.

The model consists of a Deep LSTM cell with skip connections from each input x(t) to every hidden layer, and from every hidden layer to the output y(t):

x′(t, k) = x(t)||y′(t, k - 1) , y(t) = y′(t, 1)|| …||y′(t, K)

i(t, k) = (Wₖₓᵢx′(t, k) + Wₖₕᵢh(t - 1, k) + W_(kci)c(t - 1, k) + bₖᵢ)

f(t, k) = (W_(kxf)x(t) + W_(khf)h(t - 1, k) + W_(kcf)c(t - 1, k) + b_(kf))

c(t, k) = f(t, k)c(t - 1, k) + i(t, k)tanh(W_(kxc)x′(t, k) + W_(khc)h(t - 1, k) + b_(kc))

o(t, k) = (Wₖₓₒx′(t, k) + Wₖₕₒh(t - 1, k) + W_(kco)c(t, k) + bₖₒ)

h(t, k) = o(t, k)tanh(c(t, k))

y′(t, k) = W_(kyh)(t, k) + b_(ky)

where || indicates vector concatenation, h(t, k) is the hidden state for layer k at time t, and i, f, o are the input, forget, and output gates respectively. Thus our Deep LSTM Reader is defined by g^(LSTM)(d, q) = y(|d|+|q|) with input x(t) the concatenation of d and q separated by the delimiter |||.