{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2512-14391","title":"RePo: Language Models with Context Re-Positioning","arxiv_id":"2512.14391","date":"2025-12-16","proceeding":"ICML","authors":["Huayang Li","Tianyu Zhao","Deng Cai","Richard Sproat"],"abstract":"In-context learning is fundamental to modern Large Language Models (LLMs); however, prevailing architectures impose a rigid and fixed contextual structure by assigning linear or constant positional indices. The rigid position information poses the full burden of organizing the input structure to attention layers, thus reducing the amount of attention that could be allocated for more critical information. To address this, we propose RePo, a novel mechanism that alleviates the burden for attention layers via context re-positioning. Unlike conventional approaches, RePo utilizes a differentiable module, $f_φ$, to assign token positions that capture contextual dependencies, rather than replying on pre-defined order. By continually pre-training on the OLMo-2 1B \\& 7B models, we demonstrate that RePo consistently enhances performance on tasks involving noisy contexts, structured data, and longer context length, while maintaining competitive performance on general short-context tasks. Analysis reveals that RePo successfully allocates more attention mass to distant but relevant information, assigns positions in a dense and non-linear space, and captures the intrinsic structure of the input context. Our code is at https://github.com/SakanaAI/repo.","url_abs":"https://arxiv.org/abs/2512.14391","url_pdf":"https://arxiv.org/pdf/2512.14391","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2512.14391","atlas_url":"https://app.syntology.ai/?focus=2512.14391","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2512.14391"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/SakanaAI/repo","reach":null}],"summary":{"ran":13,"unverified":5},"by_repo_kind":{"found_in_text":{"samples":18,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":18,"samples":[{"code_sha256_prefix":"4debc75bb3172a99","entry":"AccurateGELUActivation","repo":"SakanaAI/repo","repo_kind":"found_in_text","path":"OLMo/olmo/custom_modules/sequential_pe.py","file_url":"https://github.com/SakanaAI/repo/blob/HEAD/OLMo/olmo/custom_modules/sequential_pe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4debc75bb3172a99"}},{"code_sha256_prefix":"235effecfe9649db","entry":"ClassInstantier","repo":"SakanaAI/repo","repo_kind":"found_in_text","path":"OLMo/olmo/custom_modules/sequential_pe.py","file_url":"https://github.com/SakanaAI/repo/blob/HEAD/OLMo/olmo/custom_modules/sequential_pe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"235effecfe9649db"}},{"code_sha256_prefix":"88693259b070fcaf","entry":"Conv1D","repo":"SakanaAI/repo","repo_kind":"found_in_text","path":"OLMo/olmo/custom_modules/sequential_pe.py","file_url":"https://github.com/SakanaAI/repo/blob/HEAD/OLMo/olmo/custom_modules/sequential_pe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"88693259b070fcaf"}},{"code_sha256_prefix":"5cddc88126d431fe","entry":"FastGELUActivation","repo":"SakanaAI/repo","repo_kind":"found_in_text","path":"OLMo/olmo/custom_modules/sequential_pe.py","file_url":"https://github.com/SakanaAI/repo/blob/HEAD/OLMo/olmo/custom_modules/sequential_pe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5cddc88126d431fe"}},{"code_sha256_prefix":"f28cae55492a0c01","entry":"GELUActivation","repo":"SakanaAI/repo","repo_kind":"found_in_text","path":"OLMo/olmo/custom_modules/sequential_pe.py","file_url":"https://github.com/SakanaAI/repo/blob/HEAD/OLMo/olmo/custom_modules/sequential_pe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f28cae55492a0c01"}},{"code_sha256_prefix":"b4aba67d8a19bc5e","entry":"LaplaceActivation","repo":"SakanaAI/repo","repo_kind":"found_in_text","path":"OLMo/olmo/custom_modules/sequential_pe.py","file_url":"https://github.com/SakanaAI/repo/blob/HEAD/OLMo/olmo/custom_modules/sequential_pe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b4aba67d8a19bc5e"}},{"code_sha256_prefix":"279011e122526eef","entry":"LinearActivation","repo":"SakanaAI/repo","repo_kind":"found_in_text","path":"OLMo/olmo/custom_modules/sequential_pe.py","file_url":"https://github.com/SakanaAI/repo/blob/HEAD/OLMo/olmo/custom_modules/sequential_pe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"279011e122526eef"}},{"code_sha256_prefix":"e4746355d84916bc","entry":"MishActivation","repo":"SakanaAI/repo","repo_kind":"found_in_text","path":"OLMo/olmo/custom_modules/sequential_pe.py","file_url":"https://github.com/SakanaAI/repo/blob/HEAD/OLMo/olmo/custom_modules/sequential_pe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e4746355d84916bc"}},{"code_sha256_prefix":"590028faf7471d07","entry":"NewGELUActivation","repo":"SakanaAI/repo","repo_kind":"found_in_text","path":"OLMo/olmo/custom_modules/sequential_pe.py","file_url":"https://github.com/SakanaAI/repo/blob/HEAD/OLMo/olmo/custom_modules/sequential_pe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"590028faf7471d07"}},{"code_sha256_prefix":"424275bff8045cf6","entry":"PeUtils","repo":"SakanaAI/repo","repo_kind":"found_in_text","path":"OLMo/olmo/custom_modules/sequential_pe.py","file_url":"https://github.com/SakanaAI/repo/blob/HEAD/OLMo/olmo/custom_modules/sequential_pe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"424275bff8045cf6"}},{"code_sha256_prefix":"ab823319421b069d","entry":"PytorchGELUTanh","repo":"SakanaAI/repo","repo_kind":"found_in_text","path":"OLMo/olmo/custom_modules/sequential_pe.py","file_url":"https://github.com/SakanaAI/repo/blob/HEAD/OLMo/olmo/custom_modules/sequential_pe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ab823319421b069d"}},{"code_sha256_prefix":"0939fdd32fd0d09c","entry":"QuickGELUActivation","repo":"SakanaAI/repo","repo_kind":"found_in_text","path":"OLMo/olmo/custom_modules/sequential_pe.py","file_url":"https://github.com/SakanaAI/repo/blob/HEAD/OLMo/olmo/custom_modules/sequential_pe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0939fdd32fd0d09c"}},{"code_sha256_prefix":"0326a8ac0b9fe65c","entry":"ReLUSquaredActivation","repo":"SakanaAI/repo","repo_kind":"found_in_text","path":"OLMo/olmo/custom_modules/sequential_pe.py","file_url":"https://github.com/SakanaAI/repo/blob/HEAD/OLMo/olmo/custom_modules/sequential_pe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0326a8ac0b9fe65c"}},{"code_sha256_prefix":"d6a0d9471d4b71b5","entry":"SequentialPE","repo":"SakanaAI/repo","repo_kind":"found_in_text","path":"OLMo/olmo/custom_modules/sequential_pe.py","file_url":"https://github.com/SakanaAI/repo/blob/HEAD/OLMo/olmo/custom_modules/sequential_pe.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d6a0d9471d4b71b5"}},{"code_sha256_prefix":"fc3f627629efc191","entry":"SequentialPeAttention","repo":"SakanaAI/repo","repo_kind":"found_in_text","path":"OLMo/olmo/custom_modules/sequential_pe.py","file_url":"https://github.com/SakanaAI/repo/blob/HEAD/OLMo/olmo/custom_modules/sequential_pe.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fc3f627629efc191"}},{"code_sha256_prefix":"aa55ea5dd1077c8b","entry":"SequentialPeBlock","repo":"SakanaAI/repo","repo_kind":"found_in_text","path":"OLMo/olmo/custom_modules/sequential_pe.py","file_url":"https://github.com/SakanaAI/repo/blob/HEAD/OLMo/olmo/custom_modules/sequential_pe.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa55ea5dd1077c8b"}},{"code_sha256_prefix":"c773eb239da5a99c","entry":"SequentialPeMLP","repo":"SakanaAI/repo","repo_kind":"found_in_text","path":"OLMo/olmo/custom_modules/sequential_pe.py","file_url":"https://github.com/SakanaAI/repo/blob/HEAD/OLMo/olmo/custom_modules/sequential_pe.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c773eb239da5a99c"}},{"code_sha256_prefix":"39c4d7d2f08b5efa","entry":"_init_seqpe_gpt_weights","repo":"SakanaAI/repo","repo_kind":"found_in_text","path":"OLMo/olmo/custom_modules/sequential_pe.py","file_url":"https://github.com/SakanaAI/repo/blob/HEAD/OLMo/olmo/custom_modules/sequential_pe.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39c4d7d2f08b5efa"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_api"},"syntology_extracted_results":null}