{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/adaptively-sparse-transformers","title":"Adaptively Sparse Transformers","arxiv_id":"1909.00015","date":"2019-08-30","proceeding":"IJCNLP 2019 11","authors":["Gonçalo M. Correia","Vlad Niculae","André F. T. Martins"],"abstract":"Attention mechanisms have become ubiquitous in NLP. Recent architectures, notably the Transformer, learn powerful context-aware word representations through layered, multi-headed attention. The multiple heads learn diverse types of word relationships. However, with standard softmax attention, all attention heads are dense, assigning a non-zero weight to all context words. In this work, we introduce the adaptively sparse Transformer, wherein attention heads have flexible, context-dependent sparsity patterns. This sparsity is accomplished by replacing softmax with $\\alpha$-entmax: a differentiable generalization of softmax that allows low-scoring words to receive precisely zero weight. Moreover, we derive a method to automatically learn the $\\alpha$ parameter -- which controls the shape and sparsity of $\\alpha$-entmax -- allowing attention heads to choose between focused or spread-out behavior. Our adaptively sparse Transformer improves interpretability and head diversity when compared to softmax Transformers on machine translation datasets. Findings of the quantitative and qualitative analysis of our approach include that heads in different layers learn different sparsity preferences and tend to be more diverse in their attention distributions than softmax Transformers. Furthermore, at no cost in accuracy, sparsity in attention heads helps to uncover different head specializations.","url_abs":"https://arxiv.org/abs/1909.00015v2","url_pdf":"https://arxiv.org/pdf/1909.00015v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"adaptively-sparse-transformers","repo_url":"https://github.com/deep-spin/entmax","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"adaptively-sparse-transformers","repo_url":"https://github.com/prajjwal1/adaptive_transformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"adaptively-sparse-transformers","repo_url":"https://github.com/prajjwal1/fluence","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"translation","task_name":"Translation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"adaptively-sparse-transformer","method_name":"Adaptively Sparse Transformer"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"interpretability","method_name":"Interpretability"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"sparse-transformer","method_name":"Sparse Transformer"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[{"slug":"adaptively-sparse-transformer","name":"Adaptively Sparse Transformer","full_name":"Adaptively Sparse Transformer"}],"results":[{"leaderboard":"/sota/machine-translation-on-iwslt2017-german","task":"Machine Translation","dataset":"IWSLT2017 German-English","model":"Adaptively Sparse Transformer (alpha-entmax)","rank_in_archive_order":1,"of":2,"metrics":{"BLEU score":"29.9"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-iwslt2017-german","task":"Machine Translation","dataset":"IWSLT2017 German-English","model":"Adaptively Sparse Transformer (1.5-entmax)","rank_in_archive_order":2,"of":2,"metrics":{"BLEU score":"29.83"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2014-english-german","task":"Machine Translation","dataset":"WMT2014 English-German","model":"Adaptively Sparse Transformer (alpha-entmax)","rank_in_archive_order":54,"of":91,"metrics":{"BLEU score":"26.93"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2014-english-german","task":"Machine Translation","dataset":"WMT2014 English-German","model":"Adaptively Sparse Transformer (1.5-entmax)","rank_in_archive_order":66,"of":91,"metrics":{"BLEU score":"25.89"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2016-romanian","task":"Machine Translation","dataset":"WMT2016 Romanian-English","model":"Adaptively Sparse Transformer (1.5-entmax)","rank_in_archive_order":9,"of":21,"metrics":{"BLEU score":"33.1"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2016-romanian","task":"Machine Translation","dataset":"WMT2016 Romanian-English","model":"Adaptively Sparse Transformer (alpha-entmax)","rank_in_archive_order":12,"of":21,"metrics":{"BLEU score":"32.89"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1909.00015","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1909.00015"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/prajjwal1/adaptive_transformer","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/prajjwal1/fluence","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deep-spin/entmax","reach":null}],"summary":{"ran_draft_wrong":3},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1},"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e41b90069aeca727","entry":"entmax15","repo":"deep-spin/entmax","repo_kind":"official","path":"entmax/activations.py","file_url":"https://github.com/deep-spin/entmax/blob/HEAD/entmax/activations.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e41b90069aeca727"}},{"code_sha256_prefix":"8e929f00b60fd451","entry":"entmax_bisect","repo":"prajjwal1/fluence","repo_kind":"listed","path":"fluence/adaptive/entmax.py","file_url":"https://github.com/prajjwal1/fluence/blob/HEAD/fluence/adaptive/entmax.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8e929f00b60fd451"}},{"code_sha256_prefix":"df0db81c189b01e7","entry":"sparsemax","repo":"deep-spin/entmax","repo_kind":"official","path":"entmax/activations.py","file_url":"https://github.com/deep-spin/entmax/blob/HEAD/entmax/activations.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"df0db81c189b01e7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}