{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/long-context-language-modeling-with-parallel","title":"Long-Context Language Modeling with Parallel Context Encoding","arxiv_id":"2402.16617","date":"2024-02-26","proceeding":null,"authors":["Howard Yen","Tianyu Gao","Danqi Chen"],"abstract":"Extending large language models (LLMs) to process longer inputs is crucial for a wide range of applications. However, the substantial computational cost of transformers and limited generalization of positional encoding restrict the size of their context window. We introduce Context Expansion with Parallel Encoding (CEPE), a framework that can be applied to any existing decoder-only LLMs to extend their context window. CEPE employs a small encoder to process long inputs chunk by chunk, enabling the frozen decoder to utilize additional contexts via cross-attention. CEPE is efficient, generalizable, and versatile: trained with 8K-token documents, it extends the context window of LLAMA-2 to 128K tokens, offering 10x the throughput with only 1/6 of the memory. CEPE yields strong performance on language modeling and in-context learning. CEPE also excels in retrieval-augmented applications, while existing long-context models degenerate with retrieved contexts. We further introduce a CEPE variant that can extend the context window of instruction-tuned models using only unlabeled data, and showcase its effectiveness on LLAMA-2-CHAT, leading to a strong instruction-following model that can leverage very long contexts on downstream tasks.","url_abs":"https://arxiv.org/abs/2402.16617v2","url_pdf":"https://arxiv.org/pdf/2402.16617v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"long-context-language-modeling-with-parallel","repo_url":"https://github.com/princeton-nlp/cepe","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.16617","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.16617"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/princeton-nlp/cepe","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":2,"ran":3,"ran_draft_wrong":3,"unverified":4},"by_repo_kind":{"official":{"samples":12,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"princeton-nlp/cepe","repo_kind":"official","path":"modeling_llama_flash.py","file_url":"https://github.com/princeton-nlp/cepe/blob/HEAD/modeling_llama_flash.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"f725bc2d76076485","entry":"apply_rotary_pos_emb","repo":"princeton-nlp/cepe","repo_kind":"official","path":"modeling_llama_flash.py","file_url":"https://github.com/princeton-nlp/cepe/blob/HEAD/modeling_llama_flash.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f725bc2d76076485"}},{"code_sha256_prefix":"e2d9698be9ebfc0b","entry":"calibrate_nll","repo":"princeton-nlp/cepe","repo_kind":"official","path":"eval_downstream.py","file_url":"https://github.com/princeton-nlp/cepe/blob/HEAD/eval_downstream.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e2d9698be9ebfc0b"}},{"code_sha256_prefix":"6c668580324cfd97","entry":"normalize_answer","repo":"princeton-nlp/cepe","repo_kind":"official","path":"utils.py","file_url":"https://github.com/princeton-nlp/cepe/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6c668580324cfd97"}},{"code_sha256_prefix":"251deacd01d3875f","entry":"preprocess_demos","repo":"princeton-nlp/cepe","repo_kind":"official","path":"eval_downstream.py","file_url":"https://github.com/princeton-nlp/cepe/blob/HEAD/eval_downstream.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"251deacd01d3875f"}},{"code_sha256_prefix":"5df8b525aada9853","entry":"remove_citations","repo":"princeton-nlp/cepe","repo_kind":"official","path":"utils.py","file_url":"https://github.com/princeton-nlp/cepe/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5df8b525aada9853"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"princeton-nlp/cepe","repo_kind":"official","path":"modeling_llama_flash.py","file_url":"https://github.com/princeton-nlp/cepe/blob/HEAD/modeling_llama_flash.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"268c530f5ecad1f4","entry":"sample_demos","repo":"princeton-nlp/cepe","repo_kind":"official","path":"eval_downstream.py","file_url":"https://github.com/princeton-nlp/cepe/blob/HEAD/eval_downstream.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"268c530f5ecad1f4"}},{"code_sha256_prefix":"27f9fcd2b79fea52","entry":"f1_score","repo":"princeton-nlp/cepe","repo_kind":"official","path":"utils.py","file_url":"https://github.com/princeton-nlp/cepe/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"27f9fcd2b79fea52"}},{"code_sha256_prefix":"45d38779c4dcf77e","entry":"load_data","repo":"princeton-nlp/cepe","repo_kind":"official","path":"dataset_utils.py","file_url":"https://github.com/princeton-nlp/cepe/blob/HEAD/dataset_utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"45d38779c4dcf77e"}},{"code_sha256_prefix":"c1dc7ad8978154d9","entry":"load_qa_templates","repo":"princeton-nlp/cepe","repo_kind":"official","path":"dataset_utils.py","file_url":"https://github.com/princeton-nlp/cepe/blob/HEAD/dataset_utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c1dc7ad8978154d9"}},{"code_sha256_prefix":"6ec1f8f1d608c575","entry":"preprocess_alce","repo":"princeton-nlp/cepe","repo_kind":"official","path":"dataset_utils.py","file_url":"https://github.com/princeton-nlp/cepe/blob/HEAD/dataset_utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6ec1f8f1d608c575"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}