{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/continual-transformers-redundancy-free","title":"Continual Transformers: Redundancy-Free Attention for Online Inference","arxiv_id":"2201.06268","date":"2022-01-17","proceeding":null,"authors":["Lukas Hedegaard","Arian Bakhtiarnia","Alexandros Iosifidis"],"abstract":"Transformers in their common form are inherently limited to operate on whole token sequences rather than on one token at a time. Consequently, their use during online inference on time-series data entails considerable redundancy due to the overlap in successive token sequences. In this work, we propose novel formulations of the Scaled Dot-Product Attention, which enable Transformers to perform efficient online token-by-token inference on a continual input stream. Importantly, our modifications are purely to the order of computations, while the outputs and learned weights are identical to those of the original Transformer Encoder. We validate our Continual Transformer Encoder with experiments on the THUMOS14, TVSeries and GTZAN datasets with remarkable results: Our Continual one- and two-block architectures reduce the floating point operations per prediction by up to 63x and 2.6x, respectively, while retaining predictive performance.","url_abs":"https://arxiv.org/abs/2201.06268v3","url_pdf":"https://arxiv.org/pdf/2201.06268v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"continual-transformers-redundancy-free","repo_url":"https://github.com/lukashedegaard/continual-transformers","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"action-detection","task_name":"Action Detection"},{"task_slug":"audio-classification","task_name":"Audio Classification"},{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"online-action-detection","task_name":"Online Action Detection"},{"task_slug":"time-series-1","task_name":"Time Series"},{"task_slug":"time-series","task_name":"Time Series Analysis"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/online-action-detection-on-thumos-14","task":"Online Action Detection","dataset":"THUMOS'14","model":"OadTR-b2","rank_in_archive_order":9,"of":15,"metrics":{"MFLOPs per pred":"1075.7","mAP":"64.5"},"uses_additional_data":false},{"leaderboard":"/sota/online-action-detection-on-thumos-14","task":"Online Action Detection","dataset":"THUMOS'14","model":"CoOadTR-b2","rank_in_archive_order":10,"of":15,"metrics":{"MFLOPs per pred":"411.9","mAP":"64.4"},"uses_additional_data":false},{"leaderboard":"/sota/online-action-detection-on-thumos-14","task":"Online Action Detection","dataset":"THUMOS'14","model":"OadTR","rank_in_archive_order":11,"of":15,"metrics":{"MFLOPs per pred":"2513.5","mAP":"64.2"},"uses_additional_data":false},{"leaderboard":"/sota/online-action-detection-on-thumos-14","task":"Online Action Detection","dataset":"THUMOS'14","model":"OadTR-b1","rank_in_archive_order":12,"of":15,"metrics":{"MFLOPs per pred":"673","mAP":"63.9"},"uses_additional_data":false},{"leaderboard":"/sota/online-action-detection-on-thumos-14","task":"Online Action Detection","dataset":"THUMOS'14","model":"CoOadTR-b1","rank_in_archive_order":15,"of":15,"metrics":{"MFLOPs per pred":"10.6"},"uses_additional_data":false},{"leaderboard":"/sota/online-action-detection-on-tvseries","task":"Online Action Detection","dataset":"TVSeries","model":"OadTR","rank_in_archive_order":4,"of":13,"metrics":{"mCAP":"88.6"},"uses_additional_data":false},{"leaderboard":"/sota/online-action-detection-on-tvseries","task":"Online Action Detection","dataset":"TVSeries","model":"OadTR-b2","rank_in_archive_order":5,"of":13,"metrics":{"mCAP":"88.3"},"uses_additional_data":false},{"leaderboard":"/sota/online-action-detection-on-tvseries","task":"Online Action Detection","dataset":"TVSeries","model":"OadTR-b1","rank_in_archive_order":7,"of":13,"metrics":{"mCAP":"88.1"},"uses_additional_data":false},{"leaderboard":"/sota/online-action-detection-on-tvseries","task":"Online Action Detection","dataset":"TVSeries","model":"CoOadTR-b1","rank_in_archive_order":8,"of":13,"metrics":{"mCAP":"87.7"},"uses_additional_data":false},{"leaderboard":"/sota/online-action-detection-on-tvseries","task":"Online Action Detection","dataset":"TVSeries","model":"CoOadTR-b2","rank_in_archive_order":9,"of":13,"metrics":{"mCAP":"87.6"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2201.06268","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}