Papers › Improving Neural Language Models with a Continuous Cache
Improving Neural Language Models with a Continuous Cache
Edouard Grave, Armand Joulin, Nicolas Usunier
We propose an extension to neural network language models to adapt their prediction to the recent history. Our model is a simplified version of memory augmented networks, which stores past hidden activations as memory and accesses them through a dot product with the current hidden activation. This mechanism is very efficient and scales to very large memory sizes. We also draw a link between the use of external memory in neural network and cache models used with count based language models. We demonstrate on several language model datasets that our approach performs significantly better than recent memory augmented networks.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
14 repositories listed; official and paper-mentioned ones first.
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Language Modelling | WikiText-103 | Neural cache model (size = 2,000) | Test perplexity | 40.8 | #80 of 89 | Archive leaderboard | report |
| Language Modelling | WikiText-103 | Neural cache model (size = 100) | Test perplexity | 44.8 | #81 of 89 | Archive leaderboard | report |
| Language Modelling | WikiText-103 | LSTM | Test perplexity | 48.7 | #85 of 89 | Archive leaderboard | report |
| Language Modelling | WikiText-2 | Grave et al. (2016) - LSTM + continuous cache pointer | Test perplexity | 68.9 | #33 of 38 | Archive leaderboard | report |
| Language Modelling | WikiText-2 | Grave et al. (2016) - LSTM | Test perplexity | 99.3 | #37 of 38 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Introduced by this paper: Neural Cache
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections