{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improving-neural-language-models-with-a","title":"Improving Neural Language Models with a Continuous Cache","arxiv_id":"1612.04426","date":"2016-12-13","proceeding":null,"authors":["Edouard Grave","Armand Joulin","Nicolas Usunier"],"abstract":"We propose an extension to neural network language models to adapt their\nprediction to the recent history. Our model is a simplified version of memory\naugmented networks, which stores past hidden activations as memory and accesses\nthem through a dot product with the current hidden activation. This mechanism\nis very efficient and scales to very large memory sizes. We also draw a link\nbetween the use of external memory in neural network and cache models used with\ncount based language models. We demonstrate on several language model datasets\nthat our approach performs significantly better than recent memory augmented\nnetworks.","url_abs":"http://arxiv.org/abs/1612.04426v1","url_pdf":"http://arxiv.org/pdf/1612.04426v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/Asteur/RERITES-AvgWeightDescentLSTM-PoetryGeneration","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/AtheMathmo/lookahead-lstm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/SachinIchake/KALM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/ari-holtzman/genlm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/arvieFrydenlund/awd-lstm-lm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/jb33k/awd-lstm-lm-ThinkNet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/jhave/RERITES-AvgWeightDescentLSTM-PoetryGeneration","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/llppff/ptb-lstmorqrnn-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/philippwirth/awd-lstm-test","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/philippwirth/treelangrnn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/salesforce/awd-lstm-lm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/soyoung97/awd-lstm-gru","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/uclanlp/NamedEntityLanguageModel","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"improving-neural-language-models-with-a","repo_url":"https://github.com/dmlc/gluon-nlp","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mxnet","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"adagrad","method_name":"AdaGrad"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gradient-clipping","method_name":"Gradient Clipping"},{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"neural-cache","method_name":"Neural Cache"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[],"methods_introduced":[{"slug":"neural-cache","name":"Neural Cache","full_name":"Neural Cache"}],"results":[{"leaderboard":"/sota/language-modelling-on-wikitext-103","task":"Language Modelling","dataset":"WikiText-103","model":"Neural cache model (size = 2,000)","rank_in_archive_order":80,"of":89,"metrics":{"Test perplexity":"40.8"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-wikitext-103","task":"Language Modelling","dataset":"WikiText-103","model":"Neural cache model (size = 100)","rank_in_archive_order":81,"of":89,"metrics":{"Test perplexity":"44.8"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-wikitext-103","task":"Language Modelling","dataset":"WikiText-103","model":"LSTM","rank_in_archive_order":85,"of":89,"metrics":{"Test perplexity":"48.7"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-wikitext-2","task":"Language Modelling","dataset":"WikiText-2","model":"Grave et al. (2016) - LSTM + continuous cache pointer","rank_in_archive_order":33,"of":38,"metrics":{"Test perplexity":"68.9"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-wikitext-2","task":"Language Modelling","dataset":"WikiText-2","model":"Grave et al. (2016) - LSTM","rank_in_archive_order":37,"of":38,"metrics":{"Test perplexity":"99.3"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1612.04426","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}