{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/parallel-context-windows-improve-in-context","title":"Parallel Context Windows for Large Language Models","arxiv_id":"2212.10947","date":"2022-12-21","proceeding":null,"authors":["Nir Ratner","Yoav Levine","Yonatan Belinkov","Ori Ram","Inbal Magar","Omri Abend","Ehud Karpas","Amnon Shashua","Kevin Leyton-Brown","Yoav Shoham"],"abstract":"When applied to processing long text, Large Language Models (LLMs) are limited by their context window. Existing efforts to address this limitation involve training specialized architectures, and cannot be easily applied to off-the-shelf LLMs. We present Parallel Context Windows (PCW), a method that alleviates the context window restriction for any off-the-shelf LLM without further training. The key to the approach is to carve a long context into chunks (``windows''), restrict the attention mechanism to apply only within each window, and re-use the positional embeddings across the windows. Our main results test the PCW approach on in-context learning with models that range in size between 750 million and 178 billion parameters, and show substantial improvements for tasks with diverse input and output spaces. We show additional benefits in other settings where long context windows may be beneficial: multi-hop questions and retrieval-augmented question answering with multiple retrieved documents. Our results highlight Parallel Context Windows as a promising method for applying off-the-shelf LLMs in a range of settings that require long text sequences. We make our code publicly available at https://github.com/ai21labs/parallel-context-windows.","url_abs":"https://arxiv.org/abs/2212.10947v3","url_pdf":"https://arxiv.org/pdf/2212.10947v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"parallel-context-windows-improve-in-context","repo_url":"https://github.com/AI21Labs/Parallel-Context-Windows","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"2048","task_name":"Playing the Game of 2048"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2212.10947","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2212.10947"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AI21Labs/Parallel-Context-Windows","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d93aa74fbf4e4619","entry":"combine_past_key_values","repo":"AI21Labs/Parallel-Context-Windows","repo_kind":"official","path":"pcw_wrapper.py","file_url":"https://github.com/AI21Labs/Parallel-Context-Windows/blob/HEAD/pcw_wrapper.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d93aa74fbf4e4619"}},{"code_sha256_prefix":"eeddb7550894b9d8","entry":"generate_pcw_position_ids","repo":"AI21Labs/Parallel-Context-Windows","repo_kind":"official","path":"pcw_wrapper.py","file_url":"https://github.com/AI21Labs/Parallel-Context-Windows/blob/HEAD/pcw_wrapper.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eeddb7550894b9d8"}},{"code_sha256_prefix":"cbb2c50ff3858a3f","entry":"load_tokenizer","repo":"AI21Labs/Parallel-Context-Windows","repo_kind":"official","path":"model_loaders.py","file_url":"https://github.com/AI21Labs/Parallel-Context-Windows/blob/HEAD/model_loaders.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cbb2c50ff3858a3f"}},{"code_sha256_prefix":"0b86d1d91fcaeebb","entry":"n_tokens_in_prompt","repo":"AI21Labs/Parallel-Context-Windows","repo_kind":"official","path":"utils.py","file_url":"https://github.com/AI21Labs/Parallel-Context-Windows/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0b86d1d91fcaeebb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}