{"url":"/method/adaptive-masking","slug":"adaptive-masking","name":"Adaptive Masking","full_name":"Adaptive Masking","full_name_withheld":false,"description_markdown":"**Adaptive Masking** is a type of attention mechanism that allows a model to learn its own context size to attend over. For each head in [Multi-Head Attention](https://paperswithcode.com/method/multi-head-attention), a masking function is added to control for the span of the attention. A masking function is a non-increasing function that maps a\r\ndistance to a value in $\\left[0, 1\\right]$. Adaptive masking takes the following soft masking function $m\\_{z}$ parametrized by a real value $z$ in $\\left[0, S\\right]$:\r\n\r\n$$ m\\_{z}\\left(x\\right) = \\min\\left[\\max\\left[\\frac{1}{R}\\left(R+z-x\\right), 0\\right], 1\\right] $$\r\n\r\nwhere $R$ is a hyper-parameter that controls its softness. The shape of this piecewise function as a function of the distance. This soft masking function is inspired by [Jernite et al. (2017)](https://arxiv.org/abs/1611.06188). The attention weights from are then computed on the masked span:\r\n\r\n$$ a\\_{tr} = \\frac{m\\_{z}\\left(t-r\\right)\\exp\\left(s\\_{tr}\\right)}{\\sum^{t-1}\\_{q=t-S}m\\_{z}\\left(t-q\\right)\\exp\\left(s\\_{tq}\\right)}$$\r\n\r\nA $\\mathcal{l}\\_{1}$ penalization is added on the parameters $z\\_{i}$ for each attention head $i$ of the model to the loss function:\r\n\r\n$$ L = - \\log{P}\\left(w\\_{1}, \\dots, w\\_{T}\\right) + \\frac{\\lambda}{M}\\sum\\_{i}z\\_{i} $$\r\n\r\nwhere $\\lambda > 0$ is the regularization hyperparameter, and $M$ is the number of heads in each\r\nlayer. This formulation is differentiable in the parameters $z\\_{i}$, and learnt jointly with the rest of the model.","description_state":"present","introduced_year":null,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":"https://arxiv.org/abs/1905.07799v2","title":"Adaptive Attention Span in Transformers","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/facebookresearch/adaptive-span/blob/349f7a27a822803239b7a6ad9df06e1175275a64/adaptive_span.py#L17","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Attention Mechanisms","url":"/methods/category/attention-mechanisms","pwc_aliases":["attention-mechanisms-1"]}],"n_papers_tagged":31,"archive_num_papers":null,"papers_newest_first":[{"paper":null,"title":"PRETI: Patient-Aware Retinal Foundation Model via Metadata-Guided Representation Learning","date":"2025-05-18","arxiv_id":"2505.12233","n_code_links":0,"syntology":null},{"paper":null,"title":"SPAT: Sensitivity-based Multihead-attention Pruning on Time Series Forecasting Models","date":"2025-05-13","arxiv_id":"2505.08768","n_code_links":0,"syntology":null},{"paper":"/paper/high-frequency-prior-driven-adaptive-masking","title":"High-Frequency Prior-Driven Adaptive Masking for Accelerating Image Super-Resolution","date":"2025-05-11","arxiv_id":"2505.06975","n_code_links":1,"syntology":null},{"paper":"/paper/bqsched-a-non-intrusive-scheduler-for-batch","title":"BQSched: A Non-intrusive Scheduler for Batch Concurrent Queries via Reinforcement Learning","date":"2025-04-27","arxiv_id":"2504.19142","n_code_links":1,"syntology":null},{"paper":null,"title":"From Gaze to Insight: Bridging Human Visual Attention and Vision Language Model Explanation for Weakly-Supervised Medical Image Segmentation","date":"2025-04-15","arxiv_id":"2504.11368","n_code_links":0,"syntology":null},{"paper":"/paper/adaptive-rank-allocation-speeding-up-modern","title":"Adaptive Rank Allocation: Speeding Up Modern Transformers with RaNA Adapters","date":"2025-03-23","arxiv_id":"2503.18216","n_code_links":1,"syntology":{"ran":3,"of":4,"unverified":1,"pointer_only":0}},{"paper":"/paper/hac-towards-100x-compression-of-3d-gaussian","title":"HAC++: Towards 100X Compression of 3D Gaussian Splatting","date":"2025-01-21","arxiv_id":"2501.12255","n_code_links":2,"syntology":null},{"paper":null,"title":"Efficient Masked AutoEncoder for Video Object Counting and A Large-Scale Benchmark","date":"2024-11-20","arxiv_id":"2411.13056","n_code_links":0,"syntology":null},{"paper":"/paper/eh-mam-easy-to-hard-masked-acoustic-modeling","title":"EH-MAM: Easy-to-Hard Masked Acoustic Modeling for Self-Supervised Speech Representation Learning","date":"2024-10-17","arxiv_id":"2410.13179","n_code_links":1,"syntology":{"ran":6,"of":10,"unverified":4,"pointer_only":0}},{"paper":null,"title":"Adaptive Masking Enhances Visual Grounding","date":"2024-10-04","arxiv_id":"2410.03161","n_code_links":0,"syntology":null},{"paper":"/paper/pre-training-on-high-definition-x-ray-images","title":"Pre-training on High Definition X-ray Images: An Experimental Study","date":"2024-04-27","arxiv_id":"2404.17926","n_code_links":1,"syntology":null},{"paper":null,"title":"Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-training","date":"2024-04-12","arxiv_id":"2404.08327","n_code_links":0,"syntology":null},{"paper":null,"title":"Retentive Decision Transformer with Adaptive Masking for Reinforcement Learning based Recommendation Systems","date":"2024-03-26","arxiv_id":"2403.17634","n_code_links":0,"syntology":null},{"paper":"/paper/hac-hash-grid-assisted-context-for-3d","title":"HAC: Hash-grid Assisted Context for 3D Gaussian Splatting Compression","date":"2024-03-21","arxiv_id":"2403.14530","n_code_links":2,"syntology":null},{"paper":null,"title":"Exploiting Adaptive Contextual Masking for Aspect-Based Sentiment Analysis","date":"2024-02-21","arxiv_id":"2402.13722","n_code_links":0,"syntology":null},{"paper":"/paper/a-novel-hybrid-time-varying-graph-neural","title":"A novel hybrid time-varying graph neural network for traffic flow forecasting","date":"2024-01-17","arxiv_id":"2401.10155","n_code_links":0,"syntology":null},{"paper":null,"title":"Dynamic ASR Pathways: An Adaptive Masking Approach Towards Efficient Pruning of A Multilingual ASR Model","date":"2023-09-22","arxiv_id":"2309.13018","n_code_links":0,"syntology":null},{"paper":null,"title":"AMLP:Adaptive Masking Lesion Patches for Self-supervised Medical Image Segmentation","date":"2023-09-08","arxiv_id":"2309.04312","n_code_links":0,"syntology":null},{"paper":"/paper/amom-adaptive-masking-over-masking-for","title":"AMOM: Adaptive Masking over Masking for Conditional Masked Language Model","date":"2023-03-13","arxiv_id":"2303.07457","n_code_links":1,"syntology":null},{"paper":null,"title":"MPS-AMS: Masked Patches Selection and Adaptive Masking Strategy Based Self-Supervised Medical Image Segmentation","date":"2023-02-27","arxiv_id":"2302.13699","n_code_links":0,"syntology":null},{"paper":"/paper/revisiting-pre-training-in-audio-visual","title":"Revisiting Pre-training in Audio-Visual Learning","date":"2023-02-07","arxiv_id":"2302.03533","n_code_links":1,"syntology":null},{"paper":null,"title":"HAT-GAE: Self-Supervised Graph Auto-encoders with Hierarchical Adaptive Masking and Trainable Corruption","date":"2023-01-28","arxiv_id":"2301.12063","n_code_links":0,"syntology":null},{"paper":"/paper/adamae-adaptive-masking-for-efficient","title":"AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoders","date":"2022-11-16","arxiv_id":"2211.09120","n_code_links":2,"syntology":{"ran":9,"of":25,"unverified":16,"pointer_only":0}},{"paper":"/paper/student-collaboration-improves-self","title":"Multiplexed Immunofluorescence Brain Image Analysis Using Self-Supervised Dual-Loss Adaptive Masked Autoencoder","date":"2022-05-10","arxiv_id":"2205.05194","n_code_links":1,"syntology":null},{"paper":"/paper/unsupervised-representation-learning-of-2","title":"Unsupervised Representation Learning of Player Behavioral Data with Confidence Guided Masking","date":"2022-04-25","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":"/paper/generative-pre-training-from-molecules","title":"Generative Pre-Training from Molecules","date":"2021-09-16","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":"/paper/self-attention-between-datapoints-going","title":"Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning","date":"2021-06-04","arxiv_id":"2106.02584","n_code_links":3,"syntology":{"ran":10,"of":30,"unverified":20,"pointer_only":0}},{"paper":null,"title":"Uncertainty Maximization in Partially Observable Domains: A Cognitive Perspective","date":"2021-02-22","arxiv_id":"2102.11232","n_code_links":0,"syntology":null},{"paper":"/paper/improving-transformer-models-by-reordering","title":"Improving Transformer Models by Reordering their Sublayers","date":"2019-11-10","arxiv_id":"1911.03864","n_code_links":2,"syntology":null},{"paper":"/paper/augmenting-self-attention-with-persistent","title":"Augmenting Self-attention with Persistent Memory","date":"2019-07-02","arxiv_id":"1907.01470","n_code_links":2,"syntology":{"ran":5,"of":5,"unverified":0,"pointer_only":4}}],"papers_shown":30,"tasks":[{"task":"/task/language-modelling","name":"Language Modelling","papers":7},{"task":"/task/representation-learning","name":"Representation Learning","papers":7},{"task":"/task/language-modeling","name":"Language Modeling","papers":6},{"task":"/task/self-supervised-learning","name":"Self-Supervised Learning","papers":4},{"task":"/task/image-segmentation","name":"Image Segmentation","papers":3},{"task":"/task/medical-image-segmentation","name":"Medical Image Segmentation","papers":3},{"task":"/task/segmentation","name":"Segmentation","papers":3},{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":3},{"task":"/task/translation","name":"Translation","papers":3},{"task":"/task/3dgs","name":"3DGS","papers":2},{"task":"/task/attribute","name":"Attribute","papers":2},{"task":"/task/contrastive-learning","name":"Contrastive Learning","papers":2},{"task":"/task/decoder","name":"Decoder","papers":2},{"task":"/task/diagnostic","name":"Diagnostic","papers":2},{"task":"/task/feature-engineering","name":"Feature Engineering","papers":2},{"task":"/task/machine-translation","name":"Machine Translation","papers":2},{"task":"/task/novel-view-synthesis","name":"Novel View Synthesis","papers":2},{"task":"/task/quantization","name":"Quantization","papers":2},{"task":"/task/speech-recognition","name":"Speech Recognition","papers":2},{"task":"/task/speech-recognition-1","name":"speech-recognition","papers":2}],"tasks_shown":20,"n_tasks":64,"usage_by_year":[{"year":"2019","papers":3},{"year":"2021","papers":3},{"year":"2022","papers":3},{"year":"2023","papers":6},{"year":"2024","papers":9},{"year":"2025","papers":7}],"row_source":"embedded","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/adaptive-masking"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}