{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unlimiformer-long-range-transformers-with","title":"Unlimiformer: Long-Range Transformers with Unlimited Length Input","arxiv_id":"2305.01625","date":"2023-05-02","proceeding":"NeurIPS 2023 11","authors":["Amanda Bertsch","Uri Alon","Graham Neubig","Matthew R. Gormley"],"abstract":"Since the proposal of transformers, these models have been limited to bounded input lengths, because of their need to attend to every token in the input. In this work, we propose Unlimiformer: a general approach that wraps any existing pretrained encoder-decoder transformer, and offloads the cross-attention computation to a single k-nearest-neighbor (kNN) index, while the returned kNN distances are the attention dot-product scores. This kNN index can be kept on either the GPU or CPU memory and queried in sub-linear time; this way, we can index practically unlimited input sequences, while every attention head in every decoder layer retrieves its top-k keys, instead of attending to every key. We evaluate Unlimiformer on several long-document and book-summarization benchmarks, showing that it can process even 500k token-long inputs from the BookSum dataset, without any input truncation at test time. We demonstrate that Unlimiformer improves pretrained models such as BART and Longformer by extending them to unlimited inputs without additional learned weights and without modifying their code. We make our code and models publicly available at https://github.com/abertsch72/unlimiformer .","url_abs":"https://arxiv.org/abs/2305.01625v3","url_pdf":"https://arxiv.org/pdf/2305.01625v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unlimiformer-long-range-transformers-with","repo_url":"https://github.com/abertsch72/unlimiformer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"unlimiformer-long-range-transformers-with","repo_url":"https://github.com/MindSpore-scientific-2/code-14/tree/main/longformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":null,"task_name":"Book summarization"},{"task_slug":null,"task_name":"CPU"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":null,"task_name":"GPU"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"adamw","method_name":"AdamW"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bart","method_name":"BART"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"longformer","method_name":"Longformer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.01625","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.01625"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/abertsch72/unlimiformer","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindSpore-scientific-2/code-14/tree/main/longformer","reach":null}],"summary":{"ran_draft_wrong":3,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a766a95b9fd5ee9c","entry":"prepare_ctrl_input","repo":"abertsch72/unlimiformer","repo_kind":"official","path":"src/run_generation.py","file_url":"https://github.com/abertsch72/unlimiformer/blob/HEAD/src/run_generation.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a766a95b9fd5ee9c"}},{"code_sha256_prefix":"b7bf53f6b6b11a4b","entry":"prepare_xlm_input","repo":"abertsch72/unlimiformer","repo_kind":"official","path":"src/run_generation.py","file_url":"https://github.com/abertsch72/unlimiformer/blob/HEAD/src/run_generation.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b7bf53f6b6b11a4b"}},{"code_sha256_prefix":"80844fa7cfa6e8d8","entry":"prepare_xlnet_input","repo":"abertsch72/unlimiformer","repo_kind":"official","path":"src/run_generation.py","file_url":"https://github.com/abertsch72/unlimiformer/blob/HEAD/src/run_generation.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"80844fa7cfa6e8d8"}},{"code_sha256_prefix":"5cfc5db67bc70f8f","entry":"decode","repo":"abertsch72/unlimiformer","repo_kind":"official","path":"src/utils/decoding.py","file_url":"https://github.com/abertsch72/unlimiformer/blob/HEAD/src/utils/decoding.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5cfc5db67bc70f8f"}},{"code_sha256_prefix":"abbee6221df98330","entry":"drop_duplicates_in_input","repo":"abertsch72/unlimiformer","repo_kind":"official","path":"src/utils/duplicates.py","file_url":"https://github.com/abertsch72/unlimiformer/blob/HEAD/src/utils/duplicates.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"abbee6221df98330"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}