Methods › Sequential › Recurrent Neural Networks › SHA-RNN
Single Headed Attention RNN
SHA-RNN
Introduced by Stephen Merity in Single Headed Attention RNN: Stop Thinking With Your Head
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
SHA-RNN, or Single Headed Attention RNN, is a recurrent neural network, and language model when combined with an embedding input and softmax classifier, based on a core LSTM component and a single-headed attention module. Other design choices include a Boom feedforward layer and the use of layer normalization. The guiding principles of the author were to ensure simplicity in the architecture and to keep computational costs bounded (the model was originally trained with a single GPU).
Papers archive 2025-07-28
2 shown of 2, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
SHAQ: Single Headed Attention with Quasi-Recurrence 18 Aug 2021 · 0 repositories · arXiv:2108.08207
-
Single Headed Attention RNN: Stop Thinking With Your Head 26 Nov 2019 · 5 repositories · arXiv:1911.11423
Tasks archive 2025-07-28
4 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
| Task | Papers |
|---|---|
| GPU | 1 |
| Hyperparameter Optimization | 1 |
| Language Modeling | 1 |
| Language Modelling | 1 |
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections