{"url":"/method/neural-cache","slug":"neural-cache","name":"Neural Cache","full_name":"Neural Cache","full_name_withheld":false,"description_markdown":"A **Neural Cache**, or a **Continuous Cache**, is a module for language modelling which stores previous hidden states in memory cells. They are then used as keys to retrieve their corresponding word, that is the next word. There is no transformation applied to the storage during writing and reading.\r\n\r\nMore formally it exploits the hidden representations $h\\_{t}$ to define a probability distribution over the words in the cache. As\r\nillustrated in the Figure, the cache stores pairs $\\left(h\\_{i}, x\\_{i+1}\\right)$ of a hidden representation, and the word which was generated based on this representation (the vector $h\\_{i}$ encodes the history $x\\_{i}, \\dots, x\\_{1}$). At time $t$, we then define a probability distribution over words stored in the cache based on the stored hidden representations and the current one $h\\_{t}$ as:\r\n\r\n$$ p\\_{cache}\\left(w | h\\_{1\\dots{t}}, x\\_{1\\dots{t}}\\right) \\propto \\sum^{t-1}\\_{i=1}\\mathcal{1}\\_{\\text{set}\\left(w=x\\_{i+1}\\right)} \\exp\\left(θ\\_{h}>h\\_{t}^{T}h\\_{i}\\right) $$\r\n\r\nwhere the scalar $\\theta$ is a parameter which controls the flatness of the distribution. When $\\theta$ is equal to zero, the probability distribution over the history is uniform, and the model is equivalent to a unigram cache model.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Improving Neural Language Models with a Continuous Cache","paper":"/paper/improving-neural-language-models-with-a","first_author":"Edouard Grave","n_authors":3,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/improving-neural-language-models-with-a"},"source":{"url":"http://arxiv.org/abs/1612.04426v1","title":"Improving Neural Language Models with a Continuous Cache","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Natural Language Processing","area_id":"natural-language-processing","collection":"Language Model Components","url":"/methods/category/language-model-components","pwc_aliases":[]}],"n_papers_tagged":4,"archive_num_papers":4,"papers_newest_first":[{"paper":null,"title":"Information-Weighted Neural Cache Language Models for ASR","date":"2018-09-24","arxiv_id":"1809.08826","n_code_links":0,"syntology":null},{"paper":null,"title":"Neural Cache: Bit-Serial In-Cache Acceleration of Deep Neural Networks","date":"2018-05-09","arxiv_id":"1805.03718","n_code_links":0,"syntology":null},{"paper":"/paper/regularizing-and-optimizing-lstm-language","title":"Regularizing and Optimizing LSTM Language Models","date":"2017-08-07","arxiv_id":"1708.02182","n_code_links":45,"syntology":{"ran":6,"of":7,"unverified":1,"pointer_only":7}},{"paper":"/paper/improving-neural-language-models-with-a","title":"Improving Neural Language Models with a Continuous Cache","date":"2016-12-13","arxiv_id":"1612.04426","n_code_links":14,"syntology":null}],"papers_shown":4,"tasks":[{"task":"/task/language-modeling","name":"Language Modeling","papers":2},{"task":"/task/language-modelling","name":"Language Modelling","papers":2},{"task":null,"name":"CPU","papers":1},{"task":null,"name":"GPU","papers":1},{"task":"/task/image-classification","name":"Image Classification","papers":1},{"task":"/task/quantization","name":"Quantization","papers":1},{"task":"/task/translation","name":"Translation","papers":1}],"tasks_shown":7,"n_tasks":7,"usage_by_year":[{"year":"2016","papers":1},{"year":"2017","papers":1},{"year":"2018","papers":2}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/neural-cache"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}