{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2603-04759","title":"Stacked from One: Multi-Scale Self-Injection for Context Window Extension","arxiv_id":"2603.04759","date":"2026-03-05","proceeding":null,"authors":["Wei Han","Pan Zhou","Soujanya Poria","Shuicheng Yan"],"abstract":"The limited context window of contemporary large language models (LLMs) remains a primary bottleneck for their broader application across diverse domains. Although continual pre-training on long-context data offers a straightforward solution, it incurs prohibitive data acquisition and computational costs. To address this challenge, we propose~\\modelname, a novel framework based on multi-grained context compression and query-aware information acquisition. SharedLLM comprises two stacked short-context LLMs: a lower model serving as a compressor and an upper model acting as a decoder. The lower model compresses long inputs into compact, multi-grained representations, which are then forwarded to the upper model for context-aware processing. To maximize efficiency, this information transfer occurs exclusively at the lowest layers, bypassing lengthy forward passes and redundant cross-attention operations. This entire process, wherein the upper and lower models are derived from the same underlying LLM layers, is termed~\\textit{self-injection}. To support this architecture, a specialized tree-based data structure enables the efficient encoding and query-aware retrieval of contextual information. Despite being trained on sequences of only 8K tokens, \\modelname~effectively generalizes to inputs exceeding 128K tokens. Across a comprehensive suite of long-context modeling and understanding benchmarks, \\modelname~achieves performance superior or comparable to strong baselines, striking an optimal balance between efficiency and accuracy. Furthermore, these design choices allow \\modelname~to substantially reduce the memory footprint and yield notable inference speedups ($2\\times$ over streaming and $3\\times$ over encoder-decoder architectures).","url_abs":"https://arxiv.org/abs/2603.04759","url_pdf":"https://arxiv.org/pdf/2603.04759","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2603.04759","atlas_url":"https://app.syntology.ai/?focus=2603.04759","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2603.04759"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/Clement25/SharedLLM","reach":null}],"summary":{"ran_fixture":1,"ran":3,"ran_honours":1,"ran_draft_wrong":5,"ran_violates":1,"unverified":10},"by_repo_kind":{"found_in_text":{"samples":21,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":21,"samples":[{"code_sha256_prefix":"f725bc2d76076485","entry":"apply_rotary_pos_emb","repo":"Clement25/SharedLLM","repo_kind":"found_in_text","path":"modeling/modeling_llama_sharedllm_flash.py","file_url":"https://github.com/Clement25/SharedLLM/blob/HEAD/modeling/modeling_llama_sharedllm_flash.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f725bc2d76076485"}},{"code_sha256_prefix":"04b0e165da9a0405","entry":"build_chat","repo":"Clement25/SharedLLM","repo_kind":"found_in_text","path":"LongBenchTest/summ/compress.py","file_url":"https://github.com/Clement25/SharedLLM/blob/HEAD/LongBenchTest/summ/compress.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"04b0e165da9a0405"}},{"code_sha256_prefix":"b349b79d9cc2934b","entry":"count_score","repo":"Clement25/SharedLLM","repo_kind":"found_in_text","path":"LongBenchTest/metrics.py","file_url":"https://github.com/Clement25/SharedLLM/blob/HEAD/LongBenchTest/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b349b79d9cc2934b"}},{"code_sha256_prefix":"03bffbf21c24cb7b","entry":"get_word_len","repo":"Clement25/SharedLLM","repo_kind":"found_in_text","path":"LongBenchTest/summ/compress.py","file_url":"https://github.com/Clement25/SharedLLM/blob/HEAD/LongBenchTest/summ/compress.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"03bffbf21c24cb7b"}},{"code_sha256_prefix":"b80b1e9bf939544f","entry":"get_word_list","repo":"Clement25/SharedLLM","repo_kind":"found_in_text","path":"LongBenchTest/summ/compress.py","file_url":"https://github.com/Clement25/SharedLLM/blob/HEAD/LongBenchTest/summ/compress.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b80b1e9bf939544f"}},{"code_sha256_prefix":"e7e75981cb464788","entry":"normalize_answer","repo":"Clement25/SharedLLM","repo_kind":"found_in_text","path":"LongBenchTest/metrics.py","file_url":"https://github.com/Clement25/SharedLLM/blob/HEAD/LongBenchTest/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e7e75981cb464788"}},{"code_sha256_prefix":"6c668580324cfd97","entry":"normalize_answer","repo":"Clement25/SharedLLM","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/Clement25/SharedLLM/blob/HEAD/utils.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6c668580324cfd97"}},{"code_sha256_prefix":"8c5c581f9264c810","entry":"normalize_zh_answer","repo":"Clement25/SharedLLM","repo_kind":"found_in_text","path":"LongBenchTest/metrics.py","file_url":"https://github.com/Clement25/SharedLLM/blob/HEAD/LongBenchTest/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8c5c581f9264c810"}},{"code_sha256_prefix":"4489113b536ca6eb","entry":"post_process","repo":"Clement25/SharedLLM","repo_kind":"found_in_text","path":"LongBenchTest/pred.py","file_url":"https://github.com/Clement25/SharedLLM/blob/HEAD/LongBenchTest/pred.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4489113b536ca6eb"}},{"code_sha256_prefix":"5df8b525aada9853","entry":"remove_citations","repo":"Clement25/SharedLLM","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/Clement25/SharedLLM/blob/HEAD/utils.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5df8b525aada9853"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"Clement25/SharedLLM","repo_kind":"found_in_text","path":"modeling/modeling_llama_sharedllm_flash.py","file_url":"https://github.com/Clement25/SharedLLM/blob/HEAD/modeling/modeling_llama_sharedllm_flash.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"be4b1c17cda256bb","entry":"add_eos","repo":"Clement25/SharedLLM","repo_kind":"found_in_text","path":"data_beacon.py","file_url":"https://github.com/Clement25/SharedLLM/blob/HEAD/data_beacon.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"be4b1c17cda256bb"}},{"code_sha256_prefix":"e8582b3a91d38ecb","entry":"apply_rotary_pos_emb_single","repo":"Clement25/SharedLLM","repo_kind":"found_in_text","path":"modeling/modeling_llama_sharedllm_flash.py","file_url":"https://github.com/Clement25/SharedLLM/blob/HEAD/modeling/modeling_llama_sharedllm_flash.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e8582b3a91d38ecb"}},{"code_sha256_prefix":"1a27d72ff0a9a025","entry":"are_elements_of_same_length","repo":"Clement25/SharedLLM","repo_kind":"found_in_text","path":"data_beacon.py","file_url":"https://github.com/Clement25/SharedLLM/blob/HEAD/data_beacon.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1a27d72ff0a9a025"}},{"code_sha256_prefix":"9fc77302e79f7b27","entry":"build_chat","repo":"Clement25/SharedLLM","repo_kind":"found_in_text","path":"LongBenchTest/pred.py","file_url":"https://github.com/Clement25/SharedLLM/blob/HEAD/LongBenchTest/pred.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9fc77302e79f7b27"}},{"code_sha256_prefix":"27f9fcd2b79fea52","entry":"f1_score","repo":"Clement25/SharedLLM","repo_kind":"found_in_text","path":"utils.py","file_url":"https://github.com/Clement25/SharedLLM/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"27f9fcd2b79fea52"}},{"code_sha256_prefix":"74dab428120d8998","entry":"get_max_length_in_nested_lists","repo":"Clement25/SharedLLM","repo_kind":"found_in_text","path":"data_beacon.py","file_url":"https://github.com/Clement25/SharedLLM/blob/HEAD/data_beacon.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"74dab428120d8998"}},{"code_sha256_prefix":"45d38779c4dcf77e","entry":"load_data","repo":"Clement25/SharedLLM","repo_kind":"found_in_text","path":"dataset_utils.py","file_url":"https://github.com/Clement25/SharedLLM/blob/HEAD/dataset_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"45d38779c4dcf77e"}},{"code_sha256_prefix":"c1dc7ad8978154d9","entry":"load_qa_templates","repo":"Clement25/SharedLLM","repo_kind":"found_in_text","path":"dataset_utils.py","file_url":"https://github.com/Clement25/SharedLLM/blob/HEAD/dataset_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c1dc7ad8978154d9"}},{"code_sha256_prefix":"0dcb2bf78dd172f3","entry":"parse_args","repo":"Clement25/SharedLLM","repo_kind":"found_in_text","path":"LongBenchTest/pred.py","file_url":"https://github.com/Clement25/SharedLLM/blob/HEAD/LongBenchTest/pred.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0dcb2bf78dd172f3"}},{"code_sha256_prefix":"6ec1f8f1d608c575","entry":"preprocess_alce","repo":"Clement25/SharedLLM","repo_kind":"found_in_text","path":"dataset_utils.py","file_url":"https://github.com/Clement25/SharedLLM/blob/HEAD/dataset_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6ec1f8f1d608c575"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}