{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mesa-extrapolation-a-weave-position-encoding","title":"Mesa-Extrapolation: A Weave Position Encoding Method for Enhanced Extrapolation in LLMs","arxiv_id":"2410.15859","date":"2024-10-21","proceeding":null,"authors":["Xin Ma","Yang Liu","Jingjing Liu","Xiaoxu Ma"],"abstract":"Large language models (LLMs), although having revolutionized many fields, still suffer from the challenging extrapolation problem, where the inference ability of LLMs sharply declines beyond their max training lengths. In this work, we conduct a theoretical analysis to better understand why No Position Encoding (NoPE) fails outside its effective range, as well as examining the power of Position Encoding (PE) in this context. Our findings reveal that with meticulous weave position, PE can indeed be extended beyond effective range. Our theorems establish that LLMs equipped with weave PE can achieve improved extrapolation performance without additional cost. Furthermore, we introduce a novel weave PE method, Mesa-Extrapolation, which utilizes a chunk-based triangular attention matrix and applies Stair PE to manage the final chunk. This method not only retains competitive performance but also offers substantial benefits such as significantly reduced memory demand and faster inference speed. Extensive experiments validate the effectiveness of Mesa-Extrapolation, demonstrating its potential as a scalable solution to enhancing LLMs applicative reach. Our code is available at \\url{https://github.com/soacker/Mesa-Extrapolation}.","url_abs":"https://arxiv.org/abs/2410.15859v3","url_pdf":"https://arxiv.org/pdf/2410.15859v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mesa-extrapolation-a-weave-position-encoding","repo_url":"https://github.com/soacker/mesa-extrapolation","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":null,"task_name":"Position"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.15859","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.15859"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/soacker/Mesa-Extrapolation","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/soacker/mesa-extrapolation","reach":{"status":"ok"}}],"summary":{"ran":1,"unverified":4},"by_repo_kind":{"official":{"samples":5,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"cfbb7980fd327c7c","entry":"PositionSet","repo":"soacker/mesa-extrapolation","repo_kind":"official","path":"methods/mesa_extrapolation.py","file_url":"https://github.com/soacker/mesa-extrapolation/blob/HEAD/methods/mesa_extrapolation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cfbb7980fd327c7c"}},{"code_sha256_prefix":"45b84bba85a3bf23","entry":"get_data","repo":"soacker/Mesa-Extrapolation","repo_kind":"official","path":"datas/get_data.py","file_url":"https://github.com/soacker/Mesa-Extrapolation/blob/HEAD/datas/get_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"45b84bba85a3bf23"}},{"code_sha256_prefix":"f7ca92bd79b46551","entry":"get_model","repo":"soacker/Mesa-Extrapolation","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/soacker/Mesa-Extrapolation/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f7ca92bd79b46551"}},{"code_sha256_prefix":"2d6568b125252111","entry":"get_promt","repo":"soacker/Mesa-Extrapolation","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/soacker/Mesa-Extrapolation/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2d6568b125252111"}},{"code_sha256_prefix":"1b0aeb9feadc074d","entry":"get_tokenizer","repo":"soacker/Mesa-Extrapolation","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/soacker/Mesa-Extrapolation/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1b0aeb9feadc074d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}