{"url":"/method/adaptive-span-transformer","slug":"adaptive-span-transformer","name":"Adaptive Span Transformer","full_name":"Adaptive Span Transformer","full_name_withheld":false,"description_markdown":"The **Adaptive Attention Span Transformer** is a Transformer that utilises an improvement to the self-attention layer called [adaptive masking](https://paperswithcode.com/method/adaptive-masking) that allows the model to choose its own context size. This results in a network where each attention layer gathers information on their own context. This allows for scaling to input sequences of more than 8k tokens.\r\n\r\nTheir proposals are based on the observation that, with the dense attention of a traditional [Transformer](https://paperswithcode.com/method/transformer), each attention head shares the same attention span $S$ (attending over the full context). But many attention heads can specialize to more local context (others look at the longer sequence). This motivates the need for a variant of self-attention that allows the model to choose its own context size (adaptive masking - see components).","description_state":"present","introduced_year":null,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":"https://arxiv.org/abs/1905.07799v2","title":"Adaptive Attention Span in Transformers","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Natural Language Processing","area_id":"natural-language-processing","collection":"Transformers","url":"/methods/category/transformers","pwc_aliases":[]}],"n_papers_tagged":3,"archive_num_papers":null,"papers_newest_first":[{"paper":"/paper/self-attention-between-datapoints-going","title":"Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning","date":"2021-06-04","arxiv_id":"2106.02584","n_code_links":3,"syntology":{"ran":10,"of":30,"unverified":20,"pointer_only":0}},{"paper":"/paper/improving-transformer-models-by-reordering","title":"Improving Transformer Models by Reordering their Sublayers","date":"2019-11-10","arxiv_id":"1911.03864","n_code_links":2,"syntology":null},{"paper":"/paper/adaptive-attention-span-in-transformers","title":"Adaptive Attention Span in Transformers","date":"2019-05-19","arxiv_id":"1905.07799","n_code_links":8,"syntology":null}],"papers_shown":3,"tasks":[{"task":"/task/language-modeling","name":"Language Modeling","papers":2},{"task":"/task/language-modelling","name":"Language Modelling","papers":2},{"task":"/task/3d-part-segmentation","name":"3D Part Segmentation","papers":1},{"task":null,"name":"8k","papers":1},{"task":"/task/deep-learning","name":"Deep Learning","papers":1},{"task":"/task/machine-translation","name":"Machine Translation","papers":1},{"task":"/task/translation","name":"Translation","papers":1}],"tasks_shown":7,"n_tasks":7,"usage_by_year":[{"year":"2019","papers":2},{"year":"2021","papers":1}],"row_source":"embedded","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/adaptive-span-transformer"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}