{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/train-short-test-long-attention-with-linear","title":"Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation","arxiv_id":"2108.12409","date":"2021-08-27","proceeding":"ICLR 2022 4","authors":["Ofir Press","Noah A. Smith","Mike Lewis"],"abstract":"Since the introduction of the transformer model by Vaswani et al. (2017), a fundamental question has yet to be answered: how does a model achieve extrapolation at inference time for sequences that are longer than it saw during training? We first show that extrapolation can be enabled by simply changing the position representation method, though we find that current methods do not allow for efficient extrapolation. We therefore introduce a simpler and more efficient position method, Attention with Linear Biases (ALiBi). ALiBi does not add positional embeddings to word embeddings; instead, it biases query-key attention scores with a penalty that is proportional to their distance. We show that this method trains a 1.3 billion parameter model on input sequences of length 1024 that extrapolates to input sequences of length 2048, achieving the same perplexity as a sinusoidal position embedding model trained on inputs of length 2048 but training 11% faster and using 11% less memory. ALiBi's inductive bias towards recency also leads it to outperform multiple strong position methods on the WikiText-103 benchmark.","url_abs":"https://arxiv.org/abs/2108.12409v2","url_pdf":"https://arxiv.org/pdf/2108.12409v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"train-short-test-long-attention-with-linear","repo_url":"https://github.com/ofirpress/attention_with_linear_biases","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"train-short-test-long-attention-with-linear","repo_url":"https://github.com/answerdotai/modernbert","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"train-short-test-long-attention-with-linear","repo_url":"https://github.com/baichuan-inc/baichuan-13b","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"train-short-test-long-attention-with-linear","repo_url":"https://github.com/brkkaya/a_linear_attention_with_biases","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"train-short-test-long-attention-with-linear","repo_url":"https://github.com/jaketae/alibi","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"train-short-test-long-attention-with-linear","repo_url":"https://github.com/lucidrains/PaLM-jax","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null},{"paper_slug":"train-short-test-long-attention-with-linear","repo_url":"https://github.com/vinairesearch/phogpt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"paper_slug":"train-short-test-long-attention-with-linear","repo_url":"https://github.com/MindCode-4/code-6/tree/main/alibi","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"train-short-test-long-attention-with-linear","repo_url":"https://github.com/MindSpore-scientific/code-3/tree/main/alibi","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"train-short-test-long-attention-with-linear","repo_url":"https://github.com/labmlai/annotated_deep_learning_paper_implementations","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"inductive-bias","task_name":"Inductive Bias"},{"task_slug":"2048","task_name":"Playing the Game of 2048"},{"task_slug":null,"task_name":"Position"},{"task_slug":"word-embeddings","task_name":"Word Embeddings"}],"methods":[{"method_slug":"alibi","method_name":"ALiBi"}],"datasets_introduced":[],"methods_introduced":[{"slug":"alibi","name":"ALiBi","full_name":"Attention with Linear Biases"}],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2108.12409","atlas_url":"https://app.syntology.ai/?focus=2108.12409","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2108.12409"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindSpore-scientific/code-3/tree/main/alibi","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/baichuan-inc/baichuan-13b","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/brkkaya/a_linear_attention_with_biases","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lucidrains/PaLM-jax","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindCode-4/code-6/tree/main/alibi","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jaketae/alibi","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ofirpress/attention_with_linear_biases","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/ZhuiyiTechnology/roformer","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vinairesearch/phogpt","reach":{"status":"ok","spdx":"BSD-3-Clause"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/labmlai/annotated_deep_learning_paper_implementations","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/answerdotai/modernbert","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":7,"ran_honours":4,"ran_fixture":1,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":4,"ran":1,"repositories":1},"listed":{"samples":12,"ran":12,"repositories":4}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"678123248e9a2d53","entry":"ALiBiConfig","repo":"jaketae/alibi","repo_kind":"listed","path":"alibi/attention.py","file_url":"https://github.com/jaketae/alibi/blob/HEAD/alibi/attention.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"678123248e9a2d53"}},{"code_sha256_prefix":"8099385f6961bb81","entry":"ALiBiMultiHeadAttention","repo":"jaketae/alibi","repo_kind":"listed","path":"alibi/attention.py","file_url":"https://github.com/jaketae/alibi/blob/HEAD/alibi/attention.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8099385f6961bb81"}},{"code_sha256_prefix":"12ef66be25ef338d","entry":"AliBiAttention","repo":"brkkaya/a_linear_attention_with_biases","repo_kind":"listed","path":"models/alibi_attn.py","file_url":"https://github.com/brkkaya/a_linear_attention_with_biases/blob/HEAD/models/alibi_attn.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"12ef66be25ef338d"}},{"code_sha256_prefix":"201cc87ee9341bc1","entry":"BertAlibiUnpadSelfAttention","repo":"answerdotai/modernbert","repo_kind":"listed","path":"src/bert_layers/attention.py","file_url":"https://github.com/answerdotai/modernbert/blob/HEAD/src/bert_layers/attention.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"201cc87ee9341bc1"}},{"code_sha256_prefix":"06c2a55caabae8b8","entry":"IndexFirstAxis","repo":"answerdotai/modernbert","repo_kind":"listed","path":"src/bert_layers/attention.py","file_url":"https://github.com/answerdotai/modernbert/blob/HEAD/src/bert_layers/attention.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"06c2a55caabae8b8"}},{"code_sha256_prefix":"1760a9fa00a33980","entry":"IndexPutFirstAxis","repo":"answerdotai/modernbert","repo_kind":"listed","path":"src/bert_layers/attention.py","file_url":"https://github.com/answerdotai/modernbert/blob/HEAD/src/bert_layers/attention.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1760a9fa00a33980"}},{"code_sha256_prefix":"076a228889dcba7c","entry":"calc_alibi_bias","repo":"lucidrains/PaLM-jax","repo_kind":"listed","path":"palm_jax/palm_lite.py","file_url":"https://github.com/lucidrains/PaLM-jax/blob/HEAD/palm_jax/palm_lite.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"076a228889dcba7c"}},{"code_sha256_prefix":"aa08d318b01ec8ef","entry":"get_alibi_slope","repo":"jaketae/alibi","repo_kind":"listed","path":"alibi/attention.py","file_url":"https://github.com/jaketae/alibi/blob/HEAD/alibi/attention.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa08d318b01ec8ef"}},{"code_sha256_prefix":"0e4286158fe03dff","entry":"get_alibi_slopes","repo":"lucidrains/PaLM-jax","repo_kind":"listed","path":"palm_jax/palm_lite.py","file_url":"https://github.com/lucidrains/PaLM-jax/blob/HEAD/palm_jax/palm_lite.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e4286158fe03dff"}},{"code_sha256_prefix":"69e643285942e484","entry":"get_relative_positions","repo":"jaketae/alibi","repo_kind":"listed","path":"alibi/attention.py","file_url":"https://github.com/jaketae/alibi/blob/HEAD/alibi/attention.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"69e643285942e484"}},{"code_sha256_prefix":"802fa8542c8445f9","entry":"pad_input","repo":"answerdotai/modernbert","repo_kind":"listed","path":"src/bert_layers/attention.py","file_url":"https://github.com/answerdotai/modernbert/blob/HEAD/src/bert_layers/attention.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"802fa8542c8445f9"}},{"code_sha256_prefix":"82bdf940665aeeb4","entry":"safe_readline","repo":"ofirpress/attention_with_linear_biases","repo_kind":"official","path":"fairseq/binarizer.py","file_url":"https://github.com/ofirpress/attention_with_linear_biases/blob/HEAD/fairseq/binarizer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"82bdf940665aeeb4"}},{"code_sha256_prefix":"4e534e11984c06b5","entry":"unpad_input_only","repo":"answerdotai/modernbert","repo_kind":"listed","path":"src/bert_layers/attention.py","file_url":"https://github.com/answerdotai/modernbert/blob/HEAD/src/bert_layers/attention.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4e534e11984c06b5"}},{"code_sha256_prefix":"ca6cbbd64821d7fa","entry":"GatedLinear","repo":"ofirpress/attention_with_linear_biases","repo_kind":"official","path":"fairseq/modules/downsampled_multihead_attention.py","file_url":"https://github.com/ofirpress/attention_with_linear_biases/blob/HEAD/fairseq/modules/downsampled_multihead_attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"ca6cbbd64821d7fa"}},{"code_sha256_prefix":"77d359c7acb72d6a","entry":"Linear","repo":"ofirpress/attention_with_linear_biases","repo_kind":"official","path":"fairseq/modules/downsampled_multihead_attention.py","file_url":"https://github.com/ofirpress/attention_with_linear_biases/blob/HEAD/fairseq/modules/downsampled_multihead_attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"77d359c7acb72d6a"}},{"code_sha256_prefix":"1769a89f798d0556","entry":"augmented_memory","repo":"ofirpress/attention_with_linear_biases","repo_kind":"official","path":"fairseq/models/speech_to_text/modules/augmented_memory_attention.py","file_url":"https://github.com/ofirpress/attention_with_linear_biases/blob/HEAD/fairseq/models/speech_to_text/modules/augmented_memory_attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"1769a89f798d0556"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}