</> Code with Papers by Syntology

Methods › General › Attention Mechanisms › Adaptive Masking

Adaptive Masking

31 papers tagged archive 2025-07-28

archive 2025-07-28 Description, source and code snippet are the archive's method entry.

Adaptive Masking is a type of attention mechanism that allows a model to learn its own context size to attend over. For each head in Multi-Head Attention, a masking function is added to control for the span of the attention. A masking function is a non-increasing function that maps a distance to a value in [0, 1]. Adaptive masking takes the following soft masking function m_z parametrized by a real value z in [0, S]:

m_z(x) = min[max[1/R(R+z-x), 0], 1]

where R is a hyper-parameter that controls its softness. The shape of this piecewise function as a function of the distance. This soft masking function is inspired by Jernite et al. (2017). The attention weights from are then computed on the masked span:

aₜᵣ = (m_z(t-r)exp(sₜᵣ))/(∑ᵗ⁻¹_(q=t-S)m_z(t-q)exp(s_(tq)))

A 𝓁₁ penalization is added on the parameters zᵢ for each attention head i of the model to the loss function:

L = - logP(w₁, …, w_T) + λ/M∑ᵢzᵢ

where λ> 0 is the regularization hyperparameter, and M is the number of heads in each layer. This formulation is differentiable in the parameters zᵢ, and learnt jointly with the rest of the model.

Source: Adaptive Attention Span in TransformersSee Code · facebookresearch/adaptive-span

Papers archive 2025-07-28

30 shown of 31, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.

Tasks archive 2025-07-28

20 shown of 64 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.

TaskPapers
Language Modelling7
Representation Learning7
Language Modeling6
Self-Supervised Learning4
Image Segmentation3
Medical Image Segmentation3
Segmentation3
Semantic Segmentation3
Translation3
3DGS2
Attribute2
Contrastive Learning2
Decoder2
Diagnostic2
Feature Engineering2
Machine Translation2
Novel View Synthesis2
Quantization2
Speech Recognition2
speech-recognition2

Usage over time archive 2025-07-28

Papers per year tagged with Adaptive Masking: 2019 to 2025, peak 9 9 0 2019: 3 papers 2019 2020: 0 papers 2020 2021: 3 papers 2021 2022: 3 papers 2022 2023: 6 papers 2023 2024: 9 papers 2024 2025: 7 papers 2025
Papers per year the archive tags with this method, by the paper's archive date (31 dated). Bars are counts, not a trend claim.

Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).

Categories archive 2025-07-28

Attention Mechanisms

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections