{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/recurrent-context-compression-efficiently","title":"Recurrent Context Compression: Efficiently Expanding the Context Window of LLM","arxiv_id":"2406.06110","date":"2024-06-10","proceeding":null,"authors":["Chensen Huang","Guibo Zhu","Xuepeng Wang","Yifei Luo","Guojing Ge","Haoran Chen","Dong Yi","Jinqiao Wang"],"abstract":"To extend the context length of Transformer-based large language models (LLMs) and improve comprehension capabilities, we often face limitations due to computational resources and bounded memory storage capacity. This work introduces a method called Recurrent Context Compression (RCC), designed to efficiently expand the context window length of LLMs within constrained storage space. We also investigate the issue of poor model responses when both instructions and context are compressed in downstream tasks, and propose an instruction reconstruction method to mitigate this problem. We validated the effectiveness of our approach on multiple tasks, achieving a compression rate of up to 32x on text reconstruction tasks with a BLEU4 score close to 0.95, and nearly 100\\% accuracy on a passkey retrieval task with a sequence length of 1M. Finally, our method demonstrated competitive performance in long-text question-answering tasks compared to non-compressed methods, while significantly saving storage resources in long-text inference tasks. Our code, models, and demo are available at https://github.com/WUHU-G/RCC_Transformer","url_abs":"https://arxiv.org/abs/2406.06110v1","url_pdf":"https://arxiv.org/pdf/2406.06110v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"recurrent-context-compression-efficiently","repo_url":"https://github.com/WUHU-G/RCC_Transformer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"long-context-understanding","task_name":"Long-Context Understanding"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"text-compression","task_name":"Text Compression"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.06110","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.06110"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/WUHU-G/RCC_Transformer","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"96bd663092bc659c","entry":"generate_sublists","repo":"WUHU-G/RCC_Transformer","repo_kind":"official","path":"scrip/pretrain_long_mamba.py","file_url":"https://github.com/WUHU-G/RCC_Transformer/blob/HEAD/scrip/pretrain_long_mamba.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"96bd663092bc659c"}},{"code_sha256_prefix":"293e0b40e5f61121","entry":"load_and_merge_safetensors","repo":"WUHU-G/RCC_Transformer","repo_kind":"official","path":"chat_example.py","file_url":"https://github.com/WUHU-G/RCC_Transformer/blob/HEAD/chat_example.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"293e0b40e5f61121"}},{"code_sha256_prefix":"d8892cfebae56c76","entry":"make_pretrain_data_module","repo":"WUHU-G/RCC_Transformer","repo_kind":"official","path":"scrip/pretrain_long_mamba.py","file_url":"https://github.com/WUHU-G/RCC_Transformer/blob/HEAD/scrip/pretrain_long_mamba.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d8892cfebae56c76"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}