{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/can-active-memory-replace-attention","title":"Can Active Memory Replace Attention?","arxiv_id":"1610.08613","date":"2016-10-27","proceeding":"NeurIPS 2016 12","authors":["Łukasz Kaiser","Samy Bengio"],"abstract":"Several mechanisms to focus attention of a neural network on selected parts\nof its input or memory have been used successfully in deep learning models in\nrecent years. Attention has improved image classification, image captioning,\nspeech recognition, generative models, and learning algorithmic tasks, but it\nhad probably the largest impact on neural machine translation.\n  Recently, similar improvements have been obtained using alternative\nmechanisms that do not focus on a single part of a memory but operate on all of\nit in parallel, in a uniform way. Such mechanism, which we call active memory,\nimproved over attention in algorithmic tasks, image processing, and in\ngenerative modelling.\n  So far, however, active memory has not improved over attention for most\nnatural language processing tasks, in particular for machine translation. We\nanalyze this shortcoming in this paper and propose an extended model of active\nmemory that matches existing attention models on neural machine translation and\ngeneralizes better to longer sentences. We investigate this model and explain\nwhy previous active memory models did not succeed. Finally, we discuss when\nactive memory brings most benefits and where attention can be a better choice.","url_abs":"http://arxiv.org/abs/1610.08613v2","url_pdf":"http://arxiv.org/pdf/1610.08613v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"can-active-memory-replace-attention","repo_url":"https://github.com/tensorflow/models","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":null},{"paper_slug":"can-active-memory-replace-attention","repo_url":"https://github.com/tensorflow/models/tree/master/research/neural_gpu","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"translation","task_name":"Translation"},{"task_slug":"image-classification","task_name":"image-classification"},{"task_slug":"speech-recognition-1","task_name":"speech-recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/machine-translation-on-wmt2014-english-french","task":"Machine Translation","dataset":"WMT2014 English-French","model":"GRU+Attention","rank_in_archive_order":54,"of":57,"metrics":{"BLEU score":"26.4"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1610.08613","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}