Methods › Natural Language Processing › Attention Patterns
Attention Patterns
The archive attaches this collection's text per method and the copies differ: 2 distinct texts across 3 of the 3 methods here. All are shown, most-carried first (a tie goes to the text carrying Papers with Code's collection boilerplate, then to the longer text); no vote is taken between them.
Text 1, carried by 2 of 3 methods:
The original self-attention component in the Transformer architecture has a O(n²) time and memory complexity where n is the input sequence length and thus, is not efficient to scale to long inputs. Attention pattern methods look to reduce this complexity by looking at a subset of the space.
Text 2, carried by 1 of 3 methods:
Transformers are a type of neural network architecture that have several properties that make them effective for modeling data with long-range dependencies. They generally feature a combination of multi-headed attention mechanisms, residual connections, layer normalization, feedforward connections, and positional embeddings.
Also reached at /methods/category/factorized-attention (Papers with Code's slug for this collection; the archive carries no slugs, so this site's is derived from the name).
Methods
All 3 methods in this collection, most-tagged first. Year is the archive's introduced_year; the archive stores 2000 when it has none, shown here as “–”. Papers counts distinct papers the archive tags with the method. Click a heading to sort.
| Neighborhood Attention | – | 21 |
| BigBird | – | 16 |
| Routing Attention | – | 13 |