Papers › Single Headed Attention RNN: Stop Thinking With Your Head
Single Headed Attention RNN: Stop Thinking With Your Head
Stephen Merity
The leading approaches in language modeling are all obsessed with TV shows of my youth - namely Transformers and Sesame Street. Transformers this, Transformers that, and over here a bonfire worth of GPU-TPU-neuromorphic wafer scale silicon. We opt for the lazy path of old and proven techniques with a fancy crypto inspired acronym: the Single Headed Attention RNN (SHA-RNN). The author's lone goal is to show that the entire field might have evolved a different direction if we had instead been obsessed with a slightly different acronym and slightly different result. We take a previously strong language model based only on boring LSTMs and get it to within a stone's throw of a stone's throw of state-of-the-art byte level language model results on enwik8. This work has undergone no intensive hyperparameter optimization and lived entirely on a commodity desktop machine that made the author's small studio apartment far too warm in the midst of a San Franciscan summer. The final results are achievable in plus or minus 24 hours on a single GPU as the author is impatient. The attention mechanism is also readily extended to large contexts with minimal computation. Take that Sesame Street.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
1 archive task tag without a task page not shown.
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Language Modelling | enwik8 | SHA-RNN (4 layers, h=1024, attention head per layer) | Bit per Character (BPC) | 1.068 | #27 of 42 | Archive leaderboard | report |
| Language Modelling | enwik8 | SHA-RNN (4 layers, h=1024, attention head per layer) | Number of params | 54M | #27 of 42 | Archive leaderboard | report |
| Language Modelling | enwik8 | SHA-RNN (4 layers, h=1024, single attention head) | Bit per Character (BPC) | 1.076 | #28 of 42 | Archive leaderboard | report |
| Language Modelling | enwik8 | SHA-RNN (4 layers, h=1024, single attention head) | Number of params | 52M | #28 of 42 | Archive leaderboard | report |
| Language Modelling | enwik8 | SHA-LSTM (4 layers, h=1024, no attention head) | Bit per Character (BPC) | 1.33 | #39 of 42 | Archive leaderboard | report |
| Language Modelling | enwik8 | SHA-LSTM (4 layers, h=1024, no attention head) | Number of params | 51M | #39 of 42 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Introduced by this paper: SHA-RNN, Single-Headed Attention
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections