{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/llm-maybe-longlm-self-extend-llm-context","title":"LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning","arxiv_id":"2401.01325","date":"2024-01-02","proceeding":null,"authors":["Hongye Jin","Xiaotian Han","Jingfeng Yang","Zhimeng Jiang","Zirui Liu","Chia-Yuan Chang","Huiyuan Chen","Xia Hu"],"abstract":"It is well known that LLMs cannot generalize well to long contexts whose lengths are larger than the training sequence length. This poses challenges when employing LLMs for processing long input sequences during inference. In this work, we argue that LLMs themselves have inherent capabilities to handle long contexts without fine-tuning. To achieve this goal, we propose SelfExtend to extend the context window of LLMs by constructing bi-level attention information: the grouped attention and the neighbor attention. The grouped attention captures the dependencies among tokens that are far apart, while neighbor attention captures dependencies among adjacent tokens within a specified range. The two-level attentions are computed based on the original model's self-attention mechanism during inference. With minor code modification, our SelfExtend can effortlessly extend existing LLMs' context window without any fine-tuning. We conduct comprehensive experiments on multiple benchmarks and the results show that our SelfExtend can effectively extend existing LLMs' context window length. The code can be found at \\url{https://github.com/datamllab/LongLM}.","url_abs":"https://arxiv.org/abs/2401.01325v3","url_pdf":"https://arxiv.org/pdf/2401.01325v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"llm-maybe-longlm-self-extend-llm-context","repo_url":"https://github.com/datamllab/LongLM","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"llm-maybe-longlm-self-extend-llm-context","repo_url":"https://github.com/sdan/selfextend","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2401.01325","atlas_url":"https://app.syntology.ai/?focus=2401.01325","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.01325"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/datamllab/LongLM","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sdan/selfextend","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":2,"ran_violates":1,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"sdan/selfextend","repo_kind":"listed","path":"modeling_mistral.py","file_url":"https://github.com/sdan/selfextend/blob/HEAD/modeling_mistral.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"d61c483a3c2b3156","entry":"apply_rotary_pos_emb","repo":"sdan/selfextend","repo_kind":"listed","path":"modeling_mistral.py","file_url":"https://github.com/sdan/selfextend/blob/HEAD/modeling_mistral.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d61c483a3c2b3156"}},{"code_sha256_prefix":"4345de4361362772","entry":"modify_method_of_instance","repo":"datamllab/LongLM","repo_kind":"official","path":"SelfExtend.py","file_url":"https://github.com/datamllab/LongLM/blob/HEAD/SelfExtend.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4345de4361362772"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"sdan/selfextend","repo_kind":"listed","path":"modeling_mistral.py","file_url":"https://github.com/sdan/selfextend/blob/HEAD/modeling_mistral.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}