Methods › General › Attention Mechanisms › Adaptive Masking
Adaptive Masking
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Adaptive Masking is a type of attention mechanism that allows a model to learn its own context size to attend over. For each head in Multi-Head Attention, a masking function is added to control for the span of the attention. A masking function is a non-increasing function that maps a distance to a value in [0, 1]. Adaptive masking takes the following soft masking function m_z parametrized by a real value z in [0, S]:
m_z(x) = min[max[1/R(R+z-x), 0], 1]
where R is a hyper-parameter that controls its softness. The shape of this piecewise function as a function of the distance. This soft masking function is inspired by Jernite et al. (2017). The attention weights from are then computed on the masked span:
aₜᵣ = (m_z(t-r)exp(sₜᵣ))/(∑ᵗ⁻¹_(q=t-S)m_z(t-q)exp(s_(tq)))
A 𝓁₁ penalization is added on the parameters zᵢ for each attention head i of the model to the loss function:
L = - logP(w₁, …, w_T) + λ/M∑ᵢzᵢ
where λ> 0 is the regularization hyperparameter, and M is the number of heads in each layer. This formulation is differentiable in the parameters zᵢ, and learnt jointly with the rest of the model.
In Syntology code not checked for this name. The lookup matched a different name in Syntology's graph, 'masking' (resolution: token_match), and its answer is about that name only, so it does not say whether this method has code. The archive credits no paper with an arXiv id for this method, so whether this answer (about arXiv:2004.12406) is about this method is not checked.
Origin paper of this answer, arXiv:2004.12406 (its page on Syntology).
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 31 · with a code link: 17 · where Syntology ran a sample: 5 (5 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (5 of 31 tagged: 5 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 31 of 31, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
PRETI: Patient-Aware Retinal Foundation Model via Metadata-Guided Representation Learning 18 May 2025 · 0 repositories · arXiv:2505.12233
-
SPAT: Sensitivity-based Multihead-attention Pruning on Time Series Forecasting Models 13 May 2025 · 0 repositories · arXiv:2505.08768
-
High-Frequency Prior-Driven Adaptive Masking for Accelerating Image Super-Resolution 11 May 2025 · 1 repository · arXiv:2505.06975
-
BQSched: A Non-intrusive Scheduler for Batch Concurrent Queries via Reinforcement Learning 27 Apr 2025 · 1 repository · arXiv:2504.19142
-
From Gaze to Insight: Bridging Human Visual Attention and Vision Language Model Explanation for Weakly-Supervised Medical Image Segmentation 15 Apr 2025 · 0 repositories · arXiv:2504.11368
-
Adaptive Rank Allocation: Speeding Up Modern Transformers with RaNA Adapters 23 Mar 2025 · 1 repository · arXiv:2503.18216Syntology official (archive's flag): 3 ran · 3 ran (of which 1 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 4 harvested samples)
-
HAC++: Towards 100X Compression of 3D Gaussian Splatting 21 Jan 2025 · 2 repositories · arXiv:2501.12255
-
Efficient Masked AutoEncoder for Video Object Counting and A Large-Scale Benchmark 20 Nov 2024 · 0 repositories · arXiv:2411.13056
-
EH-MAM: Easy-to-Hard Masked Acoustic Modeling for Self-Supervised Speech Representation Learning 17 Oct 2024 · 1 repository · arXiv:2410.13179Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 1 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified (of 10 harvested samples) · 2 pointer-only (licence)
-
Adaptive Masking Enhances Visual Grounding 4 Oct 2024 · 0 repositories · arXiv:2410.03161
-
Pre-training on High Definition X-ray Images: An Experimental Study 27 Apr 2024 · 1 repository · arXiv:2404.17926
-
Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-training 12 Apr 2024 · 0 repositories · arXiv:2404.08327
-
Retentive Decision Transformer with Adaptive Masking for Reinforcement Learning based Recommendation Systems 26 Mar 2024 · 0 repositories · arXiv:2403.17634
-
HAC: Hash-grid Assisted Context for 3D Gaussian Splatting Compression 21 Mar 2024 · 2 repositories · arXiv:2403.14530
-
Exploiting Adaptive Contextual Masking for Aspect-Based Sentiment Analysis 21 Feb 2024 · 0 repositories · arXiv:2402.13722
-
A novel hybrid time-varying graph neural network for traffic flow forecasting 17 Jan 2024 · 0 repositories · arXiv:2401.10155
-
Dynamic ASR Pathways: An Adaptive Masking Approach Towards Efficient Pruning of A Multilingual ASR Model 22 Sep 2023 · 0 repositories · arXiv:2309.13018
-
AMLP:Adaptive Masking Lesion Patches for Self-supervised Medical Image Segmentation 8 Sep 2023 · 0 repositories · arXiv:2309.04312
-
AMOM: Adaptive Masking over Masking for Conditional Masked Language Model 13 Mar 2023 · 1 repository · arXiv:2303.07457
-
MPS-AMS: Masked Patches Selection and Adaptive Masking Strategy Based Self-Supervised Medical Image Segmentation 27 Feb 2023 · 0 repositories · arXiv:2302.13699
-
Revisiting Pre-training in Audio-Visual Learning 7 Feb 2023 · 1 repository · arXiv:2302.03533
-
HAT-GAE: Self-Supervised Graph Auto-encoders with Hierarchical Adaptive Masking and Trainable Corruption 28 Jan 2023 · 0 repositories · arXiv:2301.12063
-
AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoders 16 Nov 2022 · 2 repositories · arXiv:2211.09120Syntology official (archive's flag): 14 ran · 18 ran (of which 0 constructed an object rather than computing a result; 13 with no instrument failure: 1 honoured, 0 violated, 12 with no contract checked; 5 where Syntology's instrument failed) · 7 unverified (of 25 harvested samples) · 6 pointer-only (licence)
-
Multiplexed Immunofluorescence Brain Image Analysis Using Self-Supervised Dual-Loss Adaptive Masked Autoencoder 10 May 2022 · 1 repository · arXiv:2205.05194
-
Unsupervised Representation Learning of Player Behavioral Data with Confidence Guided Masking 25 Apr 2022 · 1 repository
-
Generative Pre-Training from Molecules 16 Sep 2021 · 1 repository
-
Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning 4 Jun 2021 · 3 repositories · arXiv:2106.02584Syntology official (archive's flag): 6 ran · 22 ran (of which 4 constructed an object rather than computing a result; 20 with no instrument failure: 2 honoured, 1 violated, 17 with no contract checked; 2 where Syntology's instrument failed) · 8 unverified (of 30 harvested samples)
-
Uncertainty Maximization in Partially Observable Domains: A Cognitive Perspective 22 Feb 2021 · 0 repositories · arXiv:2102.11232
-
Improving Transformer Models by Reordering their Sublayers 10 Nov 2019 · 2 repositories · arXiv:1911.03864
-
Augmenting Self-attention with Persistent Memory 2 Jul 2019 · 2 repositories · arXiv:1907.01470Syntology 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 3 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 5 harvested samples) · 4 pointer-only (licence)
-
Adaptive Attention Span in Transformers 19 May 2019 · 8 repositories · arXiv:1905.07799
Tasks archive 2025-07-28
20 shown of 64 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections