Methods › Natural Language Processing › Transformers › Adaptive Span Transformer
Adaptive Span Transformer
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
The Adaptive Attention Span Transformer is a Transformer that utilises an improvement to the self-attention layer called adaptive masking that allows the model to choose its own context size. This results in a network where each attention layer gathers information on their own context. This allows for scaling to input sequences of more than 8k tokens.
Their proposals are based on the observation that, with the dense attention of a traditional Transformer, each attention head shares the same attention span S (attending over the full context). But many attention heads can specialize to more local context (others look at the longer sequence). This motivates the need for a variant of self-attention that allows the model to choose its own context size (adaptive masking - see components).
Papers archive 2025-07-28
3 shown of 3, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning 4 Jun 2021 · 3 repositories · arXiv:2106.02584Syntology ran 10 of 30 samples · 20 unverified
-
Improving Transformer Models by Reordering their Sublayers 10 Nov 2019 · 2 repositories · arXiv:1911.03864
-
Adaptive Attention Span in Transformers 19 May 2019 · 8 repositories · arXiv:1905.07799
Tasks archive 2025-07-28
7 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
| Task | Papers |
|---|---|
| Language Modeling | 2 |
| Language Modelling | 2 |
| 3D Part Segmentation | 1 |
| 8k | 1 |
| Deep Learning | 1 |
| Machine Translation | 1 |
| Translation | 1 |
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections