{"url":"/method/sru-1","slug":"sru-1","name":"SRU++","full_name":"SRU++","full_name_withheld":false,"description_markdown":"**SRU++** is a self-attentive recurrent unit that combines fast recurrence and attention for sequence modeling, extending the [SRU](https://www.paperswithcode.com/method/sru) unit. The key modification of SRU++ is to incorporate more expressive non-linear operations into the recurrent network. Specifically, given the input sequence represented as a matrix $\\mathbf{X} \\in \\mathbb{R}^{L \\times d}$, the attention component computes the query, key and value representations using the following multiplications,\r\n\r\n$$\r\n\\mathbf{Q} =\\mathbf{W}^{q} \\mathbf{X}^{\\top} \r\n$$\r\n\r\n$$\r\n\\mathbf{K} =\\mathbf{W}^{k} \\mathbf{Q} \\\\\r\n$$\r\n\r\n$$\r\n\\mathbf{V} =\\mathbf{W}^{v} \\mathbf{Q}\r\n$$\r\n\r\nwhere $\\mathbf{W}^{q} \\in \\mathbb{R}^{d^{\\prime} \\times d}, \\mathbf{W}^{k}, \\mathbf{W}^{v} \\in \\mathbb{R}^{d^{\\prime} \\times d^{\\prime}}$ are model parameters. $d^{\\prime}$ is the attention dimension that is typically much smaller than $d$. Note that the keys $\\mathbf{K}$ and values $\\mathbf{V}$ are computed using $\\mathbf{Q}$ instead of $\\mathbf{X}$ such that the weight matrices $\\mathbf{W}^{k}$ and $\\mathbf{W}^{v}$ are significantly smaller. \r\n\r\nNext, we compute a weighted average output $\\mathbf{A} \\in \\mathbb{R}^{d^{\\prime} \\times L}$ using [scaled dot-product attention](https://paperswithcode.com/method/scaled):\r\n\r\n$$\r\n\\mathbf{A}^{\\top}=\\operatorname{softmax}\\left(\\frac{\\mathbf{Q}^{\\top} \\mathbf{K}}{\\sqrt{d^{\\prime}}}\\right) \\mathbf{V}^{\\top}\r\n$$\r\n\r\nThe final output $U$ required by the elementwise recurrence is obtained by another linear projection,\r\n\r\n$$\r\n\\mathbf{U}^{\\top}=\\mathbf{W}^{o}(\\mathbf{Q}+\\alpha \\cdot \\mathbf{A})\r\n$$\r\n\r\nwhere $\\alpha \\in \\mathbb{R}$ is a learned scalar and $\\mathbf{W}\\_{o} \\in \\mathbb{R}^{3 d \\times d^{\\prime}}$ is a parameter matrix. $\\mathbf{Q}+\\alpha \\cdot \\mathbf{A}$ is a [residual connection](https://paperswithcode.com/method/residual-connection) which improves gradient propagation and stabilizes training. We initialize $\\alpha$ to zero and as a result,\r\n\r\n$$\r\n\\mathbf{U}^{\\top}=\\mathbf{W}^{o} \\mathbf{Q}=\\left(\\mathbf{W}^{o} \\mathbf{W}^{q}\\right) \\mathbf{X}^{\\top}\r\n$$\r\n\r\ninitially falls back to a linear transformation of the input $X$ skipping the attention transformation. Intuitively, skipping attention encourages leveraging recurrence to capture sequential patterns during early stage of training. As $|\\alpha|$ grows, the attention mechanism can learn long-range dependencies for the model. In addition, $\\mathbf{W}^{o} \\mathbf{W}^{q}$ can be interpreted as applying a matrix factorization trick with a small inner dimension $d^{\\prime}