{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/online-and-linear-time-attention-by-enforcing","title":"Online and Linear-Time Attention by Enforcing Monotonic Alignments","arxiv_id":"1704.00784","date":"2017-04-03","proceeding":"ICML 2017 8","authors":["Colin Raffel","Minh-Thang Luong","Peter J. Liu","Ron J. Weiss","Douglas Eck"],"abstract":"Recurrent neural network models with an attention mechanism have proven to be\nextremely effective on a wide variety of sequence-to-sequence problems.\nHowever, the fact that soft attention mechanisms perform a pass over the entire\ninput sequence when producing each element in the output sequence precludes\ntheir use in online settings and results in a quadratic time complexity. Based\non the insight that the alignment between input and output sequence elements is\nmonotonic in many problems of interest, we propose an end-to-end differentiable\nmethod for learning monotonic alignments which, at test time, enables computing\nattention online and in linear time. We validate our approach on sentence\nsummarization, machine translation, and online speech recognition problems and\nachieve results competitive with existing sequence-to-sequence models.","url_abs":"http://arxiv.org/abs/1704.00784v2","url_pdf":"http://arxiv.org/pdf/1704.00784v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"online-and-linear-time-attention-by-enforcing","repo_url":"https://github.com/craffel/mad","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"online-and-linear-time-attention-by-enforcing","repo_url":"https://github.com/keonlee9420/Stepwise_Monotonic_Multihead_Attention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"sentence","task_name":"Sentence"},{"task_slug":"sentence-summarization","task_name":"Sentence Summarization"},{"task_slug":"speech-recognition","task_name":"Speech Recognition"},{"task_slug":"text-summarization","task_name":"Text Summarization"},{"task_slug":"translation","task_name":"Translation"},{"task_slug":"speech-recognition-1","task_name":"speech-recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/speech-recognition-on-timit","task":"Speech Recognition","dataset":"TIMIT","model":"Soft Monotonic Attention (ours, offline)","rank_in_archive_order":20,"of":22,"metrics":{"Percentage error":"20.1"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1704.00784","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}