Papers › Fast Parametric Learning with Activation Memorization
Fast Parametric Learning with Activation Memorization
Jack W. Rae, Chris Dyer, Peter Dayan, Timothy P. Lillicrap
Neural networks trained with backpropagation often struggle to identify classes that have been observed a small number of times. In applications where most class labels are rare, such as language modelling, this can become a performance bottleneck. One potential remedy is to augment the network with a fast-learning non-parametric model which stores recent activations and class labels into an external memory. We explore a simplified architecture where we treat a subset of the model parameters as fast memory stores. This can help retain information over longer time intervals than a traditional memory, and does not require additional space or compute. In the case of image classification, we display faster binding of novel classes on an Omniglot image curriculum task. We also show improved performance for word-based language models on news reports (GigaWord), books (Project Gutenberg) and Wikipedia articles (WikiText-103) --- the latter achieving a state-of-the-art perplexity of 29.2.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
No code repository is listed for this paper in the archive or in Syntology's graph.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Language Modelling | WikiText-103 | LSTM (Hebbian, Cache, MbPA) | Test perplexity | 29.2 | #68 of 89 | Archive leaderboard | report |
| Language Modelling | WikiText-103 | LSTM (Hebbian, Cache, MbPA) | Validation perplexity | 29.0 | #68 of 89 | Archive leaderboard | report |
| Language Modelling | WikiText-103 | LSTM (Hebbian, Cache) | Test perplexity | 29.7 | #69 of 89 | Archive leaderboard | report |
| Language Modelling | WikiText-103 | LSTM (Hebbian, Cache) | Validation perplexity | 29.9 | #69 of 89 | Archive leaderboard | report |
| Language Modelling | WikiText-103 | LSTM (Hebbian) | Test perplexity | 34.3 | #76 of 89 | Archive leaderboard | report |
| Language Modelling | WikiText-103 | LSTM (Hebbian) | Validation perplexity | 34.1 | #76 of 89 | Archive leaderboard | report |
| Language Modelling | WikiText-103 | LSTM | Test perplexity | 36.4 | #77 of 89 | Archive leaderboard | report |
| Language Modelling | WikiText-103 | LSTM | Validation perplexity | 36.0 | #77 of 89 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections