{"url":"/method/factorized-dense-synthesized-attention","slug":"factorized-dense-synthesized-attention","name":"Factorized Dense Synthesized Attention","full_name":"Factorized Dense Synthesized Attention","full_name_withheld":false,"description_markdown":"**Factorized Dense Synthesized Attention** is a synthesized attention mechanism, similar to [dense synthesized attention](https://paperswithcode.com/method/dense-synthesized-attention), but we factorize the outputs to reduce parameters and prevent overfitting. It was proposed as part of the [Synthesizer](https://paperswithcode.com/method/synthesizer) architecture. The factorized variant of the dense synthesizer can be expressed as follows:\r\n\r\n$$A, B = F\\_{A}\\left(X\\_{i}\\right), F\\_{B}\\left(X\\_{i}\\right)$$\r\n\r\nwhere $F\\_{A}\\left(.\\right)$ projects input $X\\_{i}$ into $a$ dimensions, $F\\_B\\left(.\\right)$ projects $X\\_{i}$ to $b$ dimensions, and $a \\text{ x } b = l$. The output of the factorized module is now written as:\r\n\r\n$$ Y = \\text{Softmax}\\left(C\\right)G\\left(X\\right) $$\r\n\r\nwhere $C = H\\_{A}\\left(A\\right) * H\\_{B}\\left(B\\right)$, where $H\\_{A}$, $H\\_{B}$ are tiling functions and $C \\in \\mathbb{R}^{l \\text{ x } l}$. The tiling function simply duplicates the vector $k$ times, i.e., $\\mathbb{R}^{l} \\rightarrow \\mathbb{R}^{lk}$. In this case, $H\\_{A}\\left(\\right)$ is a projection of $\\mathbb{R}^{a} \\rightarrow \\mathbb{R}^{ab}$ and $H\\_{B}\\left(\\right)$ is a projection of $\\mathbb{R}^{b} \\rightarrow \\mathbb{R}^{ba}$. To avoid having similar values within the same block, we compose the outputs of $H\\_{A}$ and $H\\_{B}$.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Synthesizer: Rethinking Self-Attention in Transformer Models","paper":"/paper/synthesizer-rethinking-self-attention-in","first_author":"Yi Tay","n_authors":6,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/synthesizer-rethinking-self-attention-in"},"source":{"url":"https://arxiv.org/abs/2005.00743v3","title":"Synthesizer: Rethinking Self-Attention in Transformer Models","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Natural Language Processing","area_id":"natural-language-processing","collection":"Synthesized Attention Mechanisms","url":"/methods/category/synthesized-attention-mechanisms","pwc_aliases":[]},{"area":"General","area_id":"general","collection":"Attention Mechanisms","url":"/methods/category/attention-mechanisms","pwc_aliases":["attention-mechanisms-1"]}],"n_papers_tagged":1,"archive_num_papers":1,"papers_newest_first":[{"paper":"/paper/synthesizer-rethinking-self-attention-in","title":"Synthesizer: Rethinking Self-Attention in Transformer Models","date":"2020-05-02","arxiv_id":"2005.00743","n_code_links":1,"syntology":{"ran":1,"of":1,"unverified":0,"pointer_only":0}}],"papers_shown":1,"tasks":[{"task":"/task/abstractive-text-summarization","name":"Abstractive Text Summarization","papers":1},{"task":"/task/dialogue-generation","name":"Dialogue Generation","papers":1},{"task":"/task/document-summarization","name":"Document Summarization","papers":1},{"task":"/task/language-modeling","name":"Language Modeling","papers":1},{"task":"/task/language-modelling","name":"Language Modelling","papers":1},{"task":"/task/linguistic-acceptability","name":"Linguistic Acceptability","papers":1},{"task":"/task/machine-translation","name":"Machine Translation","papers":1},{"task":"/task/semantic-textual-similarity","name":"Semantic Textual Similarity","papers":1},{"task":"/task/text-generation","name":"Text Generation","papers":1},{"task":"/task/translation","name":"Translation","papers":1}],"tasks_shown":10,"n_tasks":10,"usage_by_year":[{"year":"2020","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/factorized-dense-synthesized-attention"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}