Papers › Random Feature Attention
Random Feature Attention
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A. Smith, Lingpeng Kong
Transformers are state-of-the-art models for a variety of sequence modeling tasks. At their core is an attention function which models pairwise interactions between the inputs at every timestep. While attention is powerful, it does not scale efficiently to long sequences due to its quadratic time and space complexity in the sequence length. We propose RFA, a linear time and space attention that uses random feature methods to approximate the softmax function, and explore its application in transformers. RFA can be used as a drop-in replacement for conventional softmax attention and offers a straightforward way of learning with recency bias through an optional gating mechanism. Experiments on language modeling and machine translation demonstrate that RFA achieves similar or better performance compared to strong transformer baselines. In the machine translation experiment, RFA decodes twice as fast as a vanilla transformer. Compared to existing efficient transformer variants, RFA is competitive in terms of both accuracy and efficiency on three long text classification datasets. Our analysis shows that RFA's efficiency gains are especially notable on long sequences, suggesting that RFA will be particularly useful in tasks that require working with large inputs, fast decoding speed, or low memory footprints.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
No code repository is listed for this paper in the archive or in Syntology's graph.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Language Modelling | WikiText-103 | Rfa-Gate-Gaussian-Stateful (Big) | Test perplexity | 23.5 | #52 of 89 | Archive leaderboard | report |
| Language Modelling | WikiText-103 | Rfa-Gate-Gaussian-Stateful (Big) | Validation perplexity | 22 | #52 of 89 | Archive leaderboard | report |
| Language Modelling | WikiText-103 | Rfa-Gate-Gaussian-Stateful (Small) | Test perplexity | 30.5 | #70 of 89 | Archive leaderboard | report |
| Language Modelling | WikiText-103 | Rfa-Gate-Gaussian-Stateful (Small) | Validation perplexity | 29.4 | #70 of 89 | Archive leaderboard | report |
| Machine Translation | IWSLT2014 German-English | Rfa-Gate-arccos | BLEU score | 34.4 | #28 of 34 | Archive leaderboard | report |
| Machine Translation | WMT2014 English-French | Rfa-Gate-arccos | BLEU score | 39.2 | #39 of 57 | Archive leaderboard | report |
| Machine Translation | WMT2014 English-German | Rfa-Gate-arccos | BLEU score | 28.2 | #47 of 91 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections