{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/prompt-cache-modular-attention-reuse-for-low","title":"Prompt Cache: Modular Attention Reuse for Low-Latency Inference","arxiv_id":"2311.04934","date":"2023-11-07","proceeding":null,"authors":["In Gim","Guojun Chen","Seung-seob Lee","Nikhil Sarda","Anurag Khandelwal","Lin Zhong"],"abstract":"We present Prompt Cache, an approach for accelerating inference for large language models (LLM) by reusing attention states across different LLM prompts. Many input prompts have overlapping text segments, such as system messages, prompt templates, and documents provided for context. Our key insight is that by precomputing and storing the attention states of these frequently occurring text segments on the inference server, we can efficiently reuse them when these segments appear in user prompts. Prompt Cache employs a schema to explicitly define such reusable text segments, called prompt modules. The schema ensures positional accuracy during attention state reuse and provides users with an interface to access cached states in their prompt. Using a prototype implementation, we evaluate Prompt Cache across several LLMs. We show that Prompt Cache significantly reduce latency in time-to-first-token, especially for longer prompts such as document-based question answering and recommendations. The improvements range from 8x for GPU-based inference to 60x for CPU-based inference, all while maintaining output accuracy and without the need for model parameter modifications.","url_abs":"https://arxiv.org/abs/2311.04934v2","url_pdf":"https://arxiv.org/pdf/2311.04934v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"prompt-cache-modular-attention-reuse-for-low","repo_url":"https://github.com/yale-sys/prompt-cache","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":null,"task_name":"CPU"},{"task_slug":null,"task_name":"GPU"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2311.04934","atlas_url":"https://app.syntology.ai/?focus=2311.04934","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.04934"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yale-sys/prompt-cache","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"ran_honours":1,"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":10,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7e051b7017cf1d33","entry":"apply_preproc","repo":"yale-sys/prompt-cache","repo_kind":"official","path":"promptcache/prompt.py","file_url":"https://github.com/yale-sys/prompt-cache/blob/HEAD/promptcache/prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e051b7017cf1d33"}},{"code_sha256_prefix":"b349b79d9cc2934b","entry":"count_score","repo":"yale-sys/prompt-cache","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/yale-sys/prompt-cache/blob/HEAD/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b349b79d9cc2934b"}},{"code_sha256_prefix":"49917353ded2c92d","entry":"escape_tags","repo":"yale-sys/prompt-cache","repo_kind":"official","path":"benchmark/longbench.py","file_url":"https://github.com/yale-sys/prompt-cache/blob/HEAD/benchmark/longbench.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"49917353ded2c92d"}},{"code_sha256_prefix":"e7e75981cb464788","entry":"normalize_answer","repo":"yale-sys/prompt-cache","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/yale-sys/prompt-cache/blob/HEAD/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7e75981cb464788"}},{"code_sha256_prefix":"8c5c581f9264c810","entry":"normalize_zh_answer","repo":"yale-sys/prompt-cache","repo_kind":"official","path":"metrics.py","file_url":"https://github.com/yale-sys/prompt-cache/blob/HEAD/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c5c581f9264c810"}},{"code_sha256_prefix":"40430244e49148ca","entry":"pad_batch","repo":"yale-sys/prompt-cache","repo_kind":"official","path":"promptcache/cache_engine.py","file_url":"https://github.com/yale-sys/prompt-cache/blob/HEAD/promptcache/cache_engine.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"40430244e49148ca"}},{"code_sha256_prefix":"cf63debf81bfcf82","entry":"pad_unk","repo":"yale-sys/prompt-cache","repo_kind":"official","path":"promptcache/cache_engine.py","file_url":"https://github.com/yale-sys/prompt-cache/blob/HEAD/promptcache/cache_engine.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cf63debf81bfcf82"}},{"code_sha256_prefix":"09ee7b4dcf62d65d","entry":"read_file","repo":"yale-sys/prompt-cache","repo_kind":"official","path":"promptcache/prompt.py","file_url":"https://github.com/yale-sys/prompt-cache/blob/HEAD/promptcache/prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"09ee7b4dcf62d65d"}},{"code_sha256_prefix":"f116e06442e915d7","entry":"repr_indent","repo":"yale-sys/prompt-cache","repo_kind":"official","path":"promptcache/prompt.py","file_url":"https://github.com/yale-sys/prompt-cache/blob/HEAD/promptcache/prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f116e06442e915d7"}},{"code_sha256_prefix":"45ead9ded0be27ea","entry":"create_cache","repo":"yale-sys/prompt-cache","repo_kind":"official","path":"benchmark_memcpy.py","file_url":"https://github.com/yale-sys/prompt-cache/blob/HEAD/benchmark_memcpy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"45ead9ded0be27ea"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}