Methods › General › Attention Mechanisms › Random Synthesized Attention
Random Synthesized Attention
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Random Synthesized Attention is a form of synthesized attention where the attention weights are not conditioned on any input tokens. Instead, the attention weights are initialized to random values. It was introduced with the Synthesizer architecture. Random Synthesized Attention contrasts with Dense Synthesized Attention which conditions on each token independently, as opposed to pairwise token interactions in the vanilla Transformer model.
Let R be a randomly initialized matrix. Random Synthesized Attention is defined as:
Y = Softmax(R)G(X)
where R ∈ℝ^(l x l). Notably, each head adds 2 parameters to the overall network. The basic idea of the Random Synthesizer is to not rely on pairwise token interactions or any information from individual token but rather to learn a task-specific alignment that works well globally across many samples. This is a direct generalization of the recently proposed fixed self-attention patterns of Raganato et al (2020).
Papers archive 2025-07-28
1 shown of 1, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
Synthesizer: Rethinking Self-Attention in Transformer Models 2 May 2020 · 1 repository · arXiv:2005.00743Syntology ran 1 of 1 samples · 0 unverified
Tasks archive 2025-07-28
10 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections