Papers › Continual Transformers: Redundancy-Free Attention for Online Inference

Continual Transformers: Redundancy-Free Attention for Online Inference

17 Jan 2022arXiv:2201.06268archive 2025-07-28

Lukas Hedegaard, Arian Bakhtiarnia, Alexandros Iosifidis

Transformers in their common form are inherently limited to operate on whole token sequences rather than on one token at a time. Consequently, their use during online inference on time-series data entails considerable redundancy due to the overlap in successive token sequences. In this work, we propose novel formulations of the Scaled Dot-Product Attention, which enable Transformers to perform efficient online token-by-token inference on a continual input stream. Importantly, our modifications are purely to the order of computations, while the outputs and learned weights are identical to those of the original Transformer Encoder. We validate our Continual Transformer Encoder with experiments on the THUMOS14, TVSeries and GTZAN datasets with remarkable results: Our Continual one- and two-block architectures reduce the floating point operations per prediction by up to 63x and 2.6x, respectively, while retaining predictive performance.

PaperPDFCode

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

lukashedegaard/continual-transformers officialmentioned in papermentioned on GitHubpytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Action DetectionAudio ClassificationClassificationOnline Action DetectionTime SeriesTime Series Analysis

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Online Action Detection THUMOS'14 OadTR-b2 MFLOPs per pred 1075.7 #9 of 15 Archive leaderboard report
Online Action Detection THUMOS'14 OadTR-b2 mAP 64.5 #9 of 15 Archive leaderboard report
Online Action Detection THUMOS'14 CoOadTR-b2 MFLOPs per pred 411.9 #10 of 15 Archive leaderboard report
Online Action Detection THUMOS'14 CoOadTR-b2 mAP 64.4 #10 of 15 Archive leaderboard report
Online Action Detection THUMOS'14 OadTR MFLOPs per pred 2513.5 #11 of 15 Archive leaderboard report
Online Action Detection THUMOS'14 OadTR mAP 64.2 #11 of 15 Archive leaderboard report
Online Action Detection THUMOS'14 OadTR-b1 MFLOPs per pred 673 #12 of 15 Archive leaderboard report
Online Action Detection THUMOS'14 OadTR-b1 mAP 63.9 #12 of 15 Archive leaderboard report
Online Action Detection THUMOS'14 CoOadTR-b1 MFLOPs per pred 10.6 #15 of 15 Archive leaderboard report
Online Action Detection TVSeries OadTR mCAP 88.6 #4 of 13 Archive leaderboard report
Online Action Detection TVSeries OadTR-b2 mCAP 88.3 #5 of 13 Archive leaderboard report
Online Action Detection TVSeries OadTR-b1 mCAP 88.1 #7 of 13 Archive leaderboard report
Online Action Detection TVSeries CoOadTR-b1 mCAP 87.7 #8 of 13 Archive leaderboard report
Online Action Detection TVSeries CoOadTR-b2 mCAP 87.6 #9 of 13 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Absolute Position EncodingsAdamAttentionBPEDense ConnectionsDropoutLabel SmoothingLayer NormalizationLinear LayerMulti-Head AttentionPosition-Wise Feed-Forward LayerResidual ConnectionSoftmaxTransformer

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections