{"url":"/method/dense-synthesized-attention","slug":"dense-synthesized-attention","name":"Dense Synthesized Attention","full_name":"Dense Synthesized Attention","full_name_withheld":false,"description_markdown":"**Dense Synthesized Attention**, introduced with the [Synthesizer](https://paperswithcode.com/method/synthesizer) architecture, is a type of synthetic attention mechanism that replaces the notion of [query-key-values](https://paperswithcode.com/method/scaled) in the self-attention module and directly synthesizes the alignment matrix instead. Dense attention is conditioned on each input token. The method accepts an input $X \\in \\mathbb{R}^{l\\text{ x }d}$ and produces an output of $Y \\in \\mathbb{R}^{l\\text{ x }d}$. Here $l$ refers to the sequence length and $d$ refers to the dimensionality of the model. We first adopt $F\\left(.\\right)$, a parameterized function, for projecting input $X\\_{i}$ from $d$ dimensions to $l$ dimensions.\r\n\r\n$$B\\_{i} = F\\left(X\\_{i}\\right)$$\r\n\r\nwhere $F\\left(.\\right)$ is a parameterized function that maps $\\mathbb{R}^{d}$ to $\\mathbb{R}^{l}$ and $i$ is the $i$-th token of $X$. Intuitively, this can be interpreted as learning a token-wise projection to the sequence length $l$. Essentially, with this model, each token predicts weights for each token in the input sequence. In practice, a simple two layered feed-forward layer with [ReLU](https://paperswithcode.com/method/relu) activations for $F\\left(.\\right)$ is adopted:\r\n\r\n$$ F\\left(X\\right) = W\\left(\\sigma\\_{R}\\left(W(X) + b\\right)\\right) + b$$\r\n\r\nwhere $\\sigma\\_{R}$ is the ReLU activation function. Hence, $B$ is now of $\\mathbb{R}^{l\\text{ x }d}$. Given $B$, we now compute:\r\n\r\n$$ Y = \\text{Softmax}\\left(B\\right)G\\left(X\\right) $$\r\n\r\nwhere $G\\left(.\\right)$ is another parameterized function of $X$ that is analogous to $V$ (value) in the standard [Transformer](https://paperswithcode.com/method/transformer) model. This approach eliminates the [dot product](https://paperswithcode.com/method/scaled) altogether by replacing $QK^{T}$ in standard Transformers with the synthesizing function $F\\left(.\\right)$.","description_state":"present","introduced_year":null,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":"https://arxiv.org/abs/2005.00743v3","title":"Synthesizer: Rethinking Self-Attention in Transformer Models","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Attention Mechanisms","url":"/methods/category/attention-mechanisms","pwc_aliases":["attention-mechanisms-1"]}],"n_papers_tagged":1,"archive_num_papers":null,"papers_newest_first":[{"paper":"/paper/synthesizer-rethinking-self-attention-in","title":"Synthesizer: Rethinking Self-Attention in Transformer Models","date":"2020-05-02","arxiv_id":"2005.00743","n_code_links":1,"syntology":{"ran":1,"of":1,"unverified":0,"pointer_only":0}}],"papers_shown":1,"tasks":[{"task":"/task/abstractive-text-summarization","name":"Abstractive Text Summarization","papers":1},{"task":"/task/dialogue-generation","name":"Dialogue Generation","papers":1},{"task":"/task/document-summarization","name":"Document Summarization","papers":1},{"task":"/task/language-modeling","name":"Language Modeling","papers":1},{"task":"/task/language-modelling","name":"Language Modelling","papers":1},{"task":"/task/linguistic-acceptability","name":"Linguistic Acceptability","papers":1},{"task":"/task/machine-translation","name":"Machine Translation","papers":1},{"task":"/task/semantic-textual-similarity","name":"Semantic Textual Similarity","papers":1},{"task":"/task/text-generation","name":"Text Generation","papers":1},{"task":"/task/translation","name":"Translation","papers":1}],"tasks_shown":10,"n_tasks":10,"usage_by_year":[{"year":"2020","papers":1}],"row_source":"embedded","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/dense-synthesized-attention"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}