{"url":"/method/random-synthesized-attention","slug":"random-synthesized-attention","name":"Random Synthesized Attention","full_name":"Random Synthesized Attention","full_name_withheld":false,"description_markdown":"**Random Synthesized Attention** is a form of synthesized attention where the attention weights are not conditioned on any input tokens. Instead, the attention weights are initialized to random values. It was introduced with the [Synthesizer](https://paperswithcode.com/method/synthesizer) architecture. Random Synthesized Attention contrasts with [Dense Synthesized Attention](https://paperswithcode.com/method/dense-synthesized-attention) which conditions on each token independently, as opposed to pairwise token interactions in the vanilla [Transformer](https://paperswithcode.com/method/transformer) model.\r\n\r\nLet $R$ be a randomly initialized matrix. Random Synthesized Attention is defined as:\r\n\r\n$$Y = \\text{Softmax}\\left(R\\right)G\\left(X\\right) $$\r\n\r\nwhere $R \\in \\mathbb{R}^{l \\text{ x } l}$. Notably, each head adds 2 parameters to the overall network. The basic idea of the Random Synthesizer is to not rely on pairwise token interactions or any information from individual token but rather to learn a task-specific alignment that works well globally across many samples. This is a direct generalization of the recently proposed fixed self-attention patterns of [Raganato et al (2020)](https://arxiv.org/abs/2002.10260).","description_state":"present","introduced_year":null,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":"https://arxiv.org/abs/2005.00743v3","title":"Synthesizer: Rethinking Self-Attention in Transformer Models","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Attention Mechanisms","url":"/methods/category/attention-mechanisms","pwc_aliases":["attention-mechanisms-1"]}],"n_papers_tagged":1,"archive_num_papers":null,"papers_newest_first":[{"paper":"/paper/synthesizer-rethinking-self-attention-in","title":"Synthesizer: Rethinking Self-Attention in Transformer Models","date":"2020-05-02","arxiv_id":"2005.00743","n_code_links":1,"syntology":{"ran":1,"of":1,"unverified":0,"pointer_only":0}}],"papers_shown":1,"tasks":[{"task":"/task/abstractive-text-summarization","name":"Abstractive Text Summarization","papers":1},{"task":"/task/dialogue-generation","name":"Dialogue Generation","papers":1},{"task":"/task/document-summarization","name":"Document Summarization","papers":1},{"task":"/task/language-modeling","name":"Language Modeling","papers":1},{"task":"/task/language-modelling","name":"Language Modelling","papers":1},{"task":"/task/linguistic-acceptability","name":"Linguistic Acceptability","papers":1},{"task":"/task/machine-translation","name":"Machine Translation","papers":1},{"task":"/task/semantic-textual-similarity","name":"Semantic Textual Similarity","papers":1},{"task":"/task/text-generation","name":"Text Generation","papers":1},{"task":"/task/translation","name":"Translation","papers":1}],"tasks_shown":10,"n_tasks":10,"usage_by_year":[{"year":"2020","papers":1}],"row_source":"embedded","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/random-synthesized-attention"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}