{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/landmark-attention-random-access-infinite","title":"Landmark Attention: Random-Access Infinite Context Length for Transformers","arxiv_id":"2305.16300","date":"2023-05-25","proceeding":null,"authors":["Amirkeivan Mohtashami","Martin Jaggi"],"abstract":"While Transformers have shown remarkable success in natural language processing, their attention mechanism's large memory requirements have limited their ability to handle longer contexts. Prior approaches, such as recurrent memory or retrieval-based augmentation, have either compromised the random-access flexibility of attention (i.e., the capability to select any token in the entire context) or relied on separate mechanisms for relevant context retrieval, which may not be compatible with the model's attention. In this paper, we present a novel approach that allows access to the complete context while retaining random-access flexibility, closely resembling running attention on the entire context. Our method uses a landmark token to represent each block of the input and trains the attention to use it for selecting relevant blocks, enabling retrieval of blocks directly through the attention mechanism instead of by relying on a separate mechanism. Our approach seamlessly integrates with specialized data structures and the system's memory hierarchy, enabling processing of arbitrarily long context lengths. We demonstrate that our method can obtain comparable performance with Transformer-XL while significantly reducing the number of retrieved tokens in each step. Finally, we show that fine-tuning LLaMA 7B with our method successfully extends its context length capacity to over 32k tokens, allowing for inference at the context lengths of GPT-4. We release the implementation of landmark attention and the code to reproduce our experiments at https://github.com/epfml/landmark-attention/.","url_abs":"https://arxiv.org/abs/2305.16300v2","url_pdf":"https://arxiv.org/pdf/2305.16300v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"landmark-attention-random-access-infinite","repo_url":"https://github.com/epfml/landmark-attention","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"landmark-attention-random-access-infinite","repo_url":"https://github.com/cstankonrad/long_llama","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"adaptive-input-representations","method_name":"Adaptive Input Representations"},{"method_slug":"adaptive-softmax","method_name":"Adaptive Softmax"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"transformer-xl","method_name":"Transformer-XL"},{"method_slug":"variational-dropout","method_name":"Variational Dropout"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2305.16300","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.16300"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/epfml/landmark-attention","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cstankonrad/long_llama","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":5,"ran_draft_wrong":5,"ran_fixture":1,"unverified":2},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":8,"ran":7,"repositories":1},"community":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"56f315f8e37b2615","entry":"LandmarkGroupedSoftmaxFunction","repo":"epfml/landmark-attention","repo_kind":"official","path":"lm_benchmark/models/landmark.py","file_url":"https://github.com/epfml/landmark-attention/blob/HEAD/lm_benchmark/models/landmark.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"56f315f8e37b2615"}},{"code_sha256_prefix":"200bdb3c0f9ea777","entry":"LongLlamaConfig","repo":"cstankonrad/long_llama","repo_kind":"listed","path":"src/modeling_longllama.py","file_url":"https://github.com/cstankonrad/long_llama/blob/HEAD/src/modeling_longllama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"200bdb3c0f9ea777"}},{"code_sha256_prefix":"44ea22afb757bd8f","entry":"LongLlamaMemCache","repo":"cstankonrad/long_llama","repo_kind":"listed","path":"src/modeling_longllama.py","file_url":"https://github.com/cstankonrad/long_llama/blob/HEAD/src/modeling_longllama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"44ea22afb757bd8f"}},{"code_sha256_prefix":"630a906a0e052341","entry":"LongLlamaMemConfig","repo":"cstankonrad/long_llama","repo_kind":"listed","path":"src/modeling_longllama.py","file_url":"https://github.com/cstankonrad/long_llama/blob/HEAD/src/modeling_longllama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"630a906a0e052341"}},{"code_sha256_prefix":"1b00d77243e94b30","entry":"LongLlamaRotaryEmbedding","repo":"cstankonrad/long_llama","repo_kind":"listed","path":"src/modeling_longllama.py","file_url":"https://github.com/cstankonrad/long_llama/blob/HEAD/src/modeling_longllama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1b00d77243e94b30"}},{"code_sha256_prefix":"b5445674ab17410f","entry":"extract_alpaca_dataset","repo":"eugenepentland/landmark-attention-qlora","repo_kind":"community","path":"llama/train_qlora.py","file_url":"https://github.com/eugenepentland/landmark-attention-qlora/blob/HEAD/llama/train_qlora.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b5445674ab17410f"}},{"code_sha256_prefix":"f82430123a91cf21","entry":"extract_unnatural_instructions_data","repo":"eugenepentland/landmark-attention-qlora","repo_kind":"community","path":"llama/train_qlora.py","file_url":"https://github.com/eugenepentland/landmark-attention-qlora/blob/HEAD/llama/train_qlora.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f82430123a91cf21"}},{"code_sha256_prefix":"b9c4c8084f5cc40a","entry":"generate_prompt","repo":"eugenepentland/landmark-attention-qlora","repo_kind":"community","path":"llama/run_test.py","file_url":"https://github.com/eugenepentland/landmark-attention-qlora/blob/HEAD/llama/run_test.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b9c4c8084f5cc40a"}},{"code_sha256_prefix":"8dfd75aa2742a590","entry":"mem_apply_update","repo":"cstankonrad/long_llama","repo_kind":"listed","path":"src/modeling_longllama.py","file_url":"https://github.com/cstankonrad/long_llama/blob/HEAD/src/modeling_longllama.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8dfd75aa2742a590"}},{"code_sha256_prefix":"016091371cf37346","entry":"rotate_as_if_first","repo":"cstankonrad/long_llama","repo_kind":"listed","path":"src/modeling_longllama.py","file_url":"https://github.com/cstankonrad/long_llama/blob/HEAD/src/modeling_longllama.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"016091371cf37346"}},{"code_sha256_prefix":"a348555c65c6e5ff","entry":"rotate_one","repo":"cstankonrad/long_llama","repo_kind":"listed","path":"src/modeling_longllama.py","file_url":"https://github.com/cstankonrad/long_llama/blob/HEAD/src/modeling_longllama.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a348555c65c6e5ff"}},{"code_sha256_prefix":"a7aa1ee408f70207","entry":"LongLlamaAttention","repo":"cstankonrad/long_llama","repo_kind":"listed","path":"src/modeling_longllama.py","file_url":"https://github.com/cstankonrad/long_llama/blob/HEAD/src/modeling_longllama.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a7aa1ee408f70207"}},{"code_sha256_prefix":"ec8bb649218d4f43","entry":"local_dataset","repo":"eugenepentland/landmark-attention-qlora","repo_kind":"community","path":"llama/train_qlora.py","file_url":"https://github.com/eugenepentland/landmark-attention-qlora/blob/HEAD/llama/train_qlora.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ec8bb649218d4f43"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}