{"url":"/method/factorized-random-synthesized-attention","slug":"factorized-random-synthesized-attention","name":"Factorized Random Synthesized Attention","full_name":"Factorized Random Synthesized Attention","full_name_withheld":false,"description_markdown":"**Factorized Random Synthesized Attention**, introduced with the [Synthesizer](https://paperswithcode.com/method/synthesizer) architecture, is similar to [factorized dense synthesized attention](https://paperswithcode.com/method/factorized-dense-synthesized-attention) but for random synthesizers. Letting $R$ being a randomly initialized matrix, we factorize $R$ into low rank matrices $R\\_{1}, R\\_{2} \\in \\mathbb{R}^{l\\text{ x}k}$ in the attention function:\r\n\r\n$$ Y = \\text{Softmax}\\left(R\\_{1}R\\_{2}^{T}\\right)G\\left(X\\right) . $$\r\n\r\nHere $G\\left(.\\right)$ is a parameterized function that is equivalent to $V$ in [Scaled Dot-Product Attention](https://paperswithcode.com/method/scaled).\r\n\r\nFor each head, the factorization reduces the parameter costs from $l^{2}$ to $2\\left(lk\\right)$ where\r\n$k << l$ and hence helps prevent overfitting. In practice, we use a small value of $k = 8$.\r\n\r\nThe basic idea of a  Random Synthesizer is to not rely on pairwise token interactions or any information from individual token but rather to learn a task-specific alignment that works well globally across many samples.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Synthesizer: Rethinking Self-Attention in Transformer Models","paper":"/paper/synthesizer-rethinking-self-attention-in","first_author":"Yi Tay","n_authors":6,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/synthesizer-rethinking-self-attention-in"},"source":{"url":"https://arxiv.org/abs/2005.00743v3","title":"Synthesizer: Rethinking Self-Attention in Transformer Models","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Natural Language Processing","area_id":"natural-language-processing","collection":"Synthesized Attention Mechanisms","url":"/methods/category/synthesized-attention-mechanisms","pwc_aliases":[]},{"area":"General","area_id":"general","collection":"Attention Mechanisms","url":"/methods/category/attention-mechanisms","pwc_aliases":["attention-mechanisms-1"]}],"n_papers_tagged":1,"archive_num_papers":1,"papers_newest_first":[{"paper":"/paper/synthesizer-rethinking-self-attention-in","title":"Synthesizer: Rethinking Self-Attention in Transformer Models","date":"2020-05-02","arxiv_id":"2005.00743","n_code_links":1,"syntology":{"ran":1,"of":1,"unverified":0,"pointer_only":0}}],"papers_shown":1,"tasks":[{"task":"/task/abstractive-text-summarization","name":"Abstractive Text Summarization","papers":1},{"task":"/task/dialogue-generation","name":"Dialogue Generation","papers":1},{"task":"/task/document-summarization","name":"Document Summarization","papers":1},{"task":"/task/language-modeling","name":"Language Modeling","papers":1},{"task":"/task/language-modelling","name":"Language Modelling","papers":1},{"task":"/task/linguistic-acceptability","name":"Linguistic Acceptability","papers":1},{"task":"/task/machine-translation","name":"Machine Translation","papers":1},{"task":"/task/semantic-textual-similarity","name":"Semantic Textual Similarity","papers":1},{"task":"/task/text-generation","name":"Text Generation","papers":1},{"task":"/task/translation","name":"Translation","papers":1}],"tasks_shown":10,"n_tasks":10,"usage_by_year":[{"year":"2020","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/factorized-random-synthesized-attention"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}