Methods › General › Attention Modules › All-Attention Layer
All-Attention Layer
Introduced by Sainbayar Sukhbaatar et al. in Augmenting Self-attention with Persistent Memory
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
An All-Attention Layer is an attention module and layer for transformers that merges the self-attention and feedforward sublayers into a single unified attention layer. As opposed to the two-step mechanism of the Transformer layer, it directly builds its representation from the context and a persistent memory block without going through a feedforward transformation. The additional persistent memory block stores, in the form of key-value vectors, information that does not depend on the context. In terms of parameters, these persistent key-value vectors replace the feedforward sublayer.
Papers archive 2025-07-28
2 shown of 2, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
Solving Dynamic Traveling Salesman Problems With Deep Reinforcement Learning 1 Apr 2023 · 1 repository
-
Augmenting Self-attention with Persistent Memory 2 Jul 2019 · 2 repositories · arXiv:1907.01470Syntology ran 5 of 5 samples · 0 unverified · 4 pointer-only (licence)
Tasks archive 2025-07-28
7 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
| Task | Papers |
|---|---|
| Deep Reinforcement Learning | 1 |
| Language Modeling | 1 |
| Language Modelling | 1 |
| Reinforcement Learning | 1 |
| Translation | 1 |
| Traveling Salesman Problem | 1 |
| reinforcement-learning | 1 |
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections