{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fast-parametric-learning-with-activation","title":"Fast Parametric Learning with Activation Memorization","arxiv_id":"1803.10049","date":"2018-03-27","proceeding":"ICML 2018 7","authors":["Jack W. Rae","Chris Dyer","Peter Dayan","Timothy P. Lillicrap"],"abstract":"Neural networks trained with backpropagation often struggle to identify\nclasses that have been observed a small number of times. In applications where\nmost class labels are rare, such as language modelling, this can become a\nperformance bottleneck. One potential remedy is to augment the network with a\nfast-learning non-parametric model which stores recent activations and class\nlabels into an external memory. We explore a simplified architecture where we\ntreat a subset of the model parameters as fast memory stores. This can help\nretain information over longer time intervals than a traditional memory, and\ndoes not require additional space or compute. In the case of image\nclassification, we display faster binding of novel classes on an Omniglot image\ncurriculum task. We also show improved performance for word-based language\nmodels on news reports (GigaWord), books (Project Gutenberg) and Wikipedia\narticles (WikiText-103) --- the latter achieving a state-of-the-art perplexity\nof 29.2.","url_abs":"http://arxiv.org/abs/1803.10049v1","url_pdf":"http://arxiv.org/pdf/1803.10049v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"articles","task_name":"Articles"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"memorization","task_name":"Memorization"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/language-modelling-on-wikitext-103","task":"Language Modelling","dataset":"WikiText-103","model":"LSTM (Hebbian, Cache, MbPA)","rank_in_archive_order":68,"of":89,"metrics":{"Test perplexity":"29.2","Validation perplexity":"29.0"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-wikitext-103","task":"Language Modelling","dataset":"WikiText-103","model":"LSTM (Hebbian, Cache)","rank_in_archive_order":69,"of":89,"metrics":{"Test perplexity":"29.7","Validation perplexity":"29.9"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-wikitext-103","task":"Language Modelling","dataset":"WikiText-103","model":"LSTM (Hebbian)","rank_in_archive_order":76,"of":89,"metrics":{"Test perplexity":"34.3","Validation perplexity":"34.1"},"uses_additional_data":false},{"leaderboard":"/sota/language-modelling-on-wikitext-103","task":"Language Modelling","dataset":"WikiText-103","model":"LSTM","rank_in_archive_order":77,"of":89,"metrics":{"Test perplexity":"36.4","Validation perplexity":"36.0"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1803.10049","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}