{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pose-efficient-context-window-extension-of","title":"PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training","arxiv_id":"2309.10400","date":"2023-09-19","proceeding":null,"authors":["Dawei Zhu","Nan Yang","Liang Wang","YiFan Song","Wenhao Wu","Furu Wei","Sujian Li"],"abstract":"Large Language Models (LLMs) are trained with a pre-defined context length, restricting their use in scenarios requiring long inputs. Previous efforts for adapting LLMs to a longer length usually requires fine-tuning with this target length (Full-length fine-tuning), suffering intensive training cost. To decouple train length from target length for efficient context window extension, we propose Positional Skip-wisE (PoSE) training that smartly simulates long inputs using a fixed context window. This is achieved by first dividing the original context window into several chunks, then designing distinct skipping bias terms to manipulate the position indices of each chunk. These bias terms and the lengths of each chunk are altered for every training example, allowing the model to adapt to all positions within target length. Experimental results show that PoSE greatly reduces memory and time overhead compared with Full-length fine-tuning, with minimal impact on performance. Leveraging this advantage, we have successfully extended the LLaMA model to 128k tokens using a 2k training context window. Furthermore, we empirically confirm that PoSE is compatible with all RoPE-based LLMs and position interpolation strategies. Notably, our method can potentially support infinite length, limited only by memory usage in inference. With ongoing progress for efficient inference, we believe PoSE can further scale the context window beyond 128k.","url_abs":"https://arxiv.org/abs/2309.10400v3","url_pdf":"https://arxiv.org/pdf/2309.10400v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pose-efficient-context-window-extension-of","repo_url":"https://github.com/dwzhu-pku/pose","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"pose-efficient-context-window-extension-of","repo_url":"https://github.com/jzhang38/EasyContext","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"2k","task_name":"2k"},{"task_slug":null,"task_name":"Position"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.10400","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.10400"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dwzhu-pku/pose","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jzhang38/EasyContext","reach":null}],"summary":{"ran_draft_wrong":4},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b1e7a91e8f004527","entry":"train_preprocess_function_pi","repo":"dwzhu-pku/pose","repo_kind":"official","path":"src/train_pose.py","file_url":"https://github.com/dwzhu-pku/pose/blob/HEAD/src/train_pose.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1e7a91e8f004527"}},{"code_sha256_prefix":"bf446254dc4125e5","entry":"train_preprocess_function_pose","repo":"dwzhu-pku/pose","repo_kind":"official","path":"src/train_pose.py","file_url":"https://github.com/dwzhu-pku/pose/blob/HEAD/src/train_pose.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bf446254dc4125e5"}},{"code_sha256_prefix":"694357d45a3a5b27","entry":"train_preprocess_function_pose","repo":"dwzhu-pku/pose","repo_kind":"official","path":"src/train_pose.py","file_url":"https://github.com/dwzhu-pku/pose/blob/HEAD/src/train_pose.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"694357d45a3a5b27"}},{"code_sha256_prefix":"13ff067830de686c","entry":"train_preprocess_function_randomized","repo":"dwzhu-pku/pose","repo_kind":"official","path":"src/train_pose.py","file_url":"https://github.com/dwzhu-pku/pose/blob/HEAD/src/train_pose.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"13ff067830de686c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}