{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unbounded-cache-model-for-online-language","title":"Unbounded cache model for online language modeling with open vocabulary","arxiv_id":"1711.02604","date":"2017-11-07","proceeding":"NeurIPS 2017 12","authors":["Edouard Grave","Moustapha Cisse","Armand Joulin"],"abstract":"Recently, continuous cache models were proposed as extensions to recurrent\nneural network language models, to adapt their predictions to local changes in\nthe data distribution. These models only capture the local context, of up to a\nfew thousands tokens. In this paper, we propose an extension of continuous\ncache models, which can scale to larger contexts. In particular, we use a large\nscale non-parametric memory component that stores all the hidden activations\nseen in the past. We leverage recent advances in approximate nearest neighbor\nsearch and quantization algorithms to store millions of representations while\nsearching them efficiently. We conduct extensive experiments showing that our\napproach significantly improves the perplexity of pre-trained language models\non new distributions, and can scale efficiently to much larger contexts than\npreviously proposed local cache models.","url_abs":"http://arxiv.org/abs/1711.02604v1","url_pdf":"http://arxiv.org/pdf/1711.02604v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unbounded-cache-model-for-online-language","repo_url":"https://github.com/CoderINusE/NIPS-implementation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"unbounded-cache-model-for-online-language","repo_url":"https://github.com/CoderINusE/unbounded-cache-lm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1711.02604","atlas_url":"https://app.syntology.ai/?focus=1711.02604","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}