{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/do-large-language-models-need-a-content","title":"Do Large Language Models Need a Content Delivery Network?","arxiv_id":"2409.13761","date":"2024-09-16","proceeding":null,"authors":["Yihua Cheng","Kuntai Du","Jiayi Yao","Junchen Jiang"],"abstract":"As the use of large language models (LLMs) expands rapidly, so does the range of knowledge needed to supplement various LLM queries. Thus, enabling flexible and efficient injection of new knowledge in LLM inference is critical. Three high-level options exist: (i) embedding the knowledge in LLM's weights (i.e., fine-tuning), (ii) including the knowledge as a part of LLM's text input (i.e., in-context learning), or (iii) injecting the KV caches of the new knowledge to LLM during prefill. This paper argues that, although fine-tuning and in-context learning are popular, using KV caches as the medium of knowledge could simultaneously enable more modular management of knowledge injection and more efficient LLM serving with low cost and fast response. To realize these benefits, we envision a Knowledge Delivery Network (KDN), a new system component in LLM services that dynamically optimizes the storage, transfer, and composition of KV cache across LLM engines and other compute and storage resources. We believe that, just like content delivery networks (CDNs), such as Akamai, enabled the success of the Internet ecosystem through their efficient data delivery, KDNs will be critical to the success of LLM applications through their efficient knowledge delivery. We have open-sourced a KDN prototype at https://github.com/LMCache/LMCache.","url_abs":"https://arxiv.org/abs/2409.13761v2","url_pdf":"https://arxiv.org/pdf/2409.13761v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"do-large-language-models-need-a-content","repo_url":"https://github.com/lmcache/lmcache","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.13761","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.13761"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lmcache/lmcache","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"10a8aa5b7baccd39","entry":"default_output_path","repo":"lmcache/lmcache","repo_kind":"official","path":"lmcache/cli/profiling.py","file_url":"https://github.com/lmcache/lmcache/blob/HEAD/lmcache/cli/profiling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"10a8aa5b7baccd39"}},{"code_sha256_prefix":"eda16f8e754cd24d","entry":"init_logger","repo":"lmcache/lmcache","repo_kind":"official","path":"lmcache/logging.py","file_url":"https://github.com/lmcache/lmcache/blob/HEAD/lmcache/logging.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eda16f8e754cd24d"}},{"code_sha256_prefix":"eddda82f46c6b0de","entry":"normalize_url","repo":"lmcache/lmcache","repo_kind":"official","path":"lmcache/cli/http.py","file_url":"https://github.com/lmcache/lmcache/blob/HEAD/lmcache/cli/http.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eddda82f46c6b0de"}},{"code_sha256_prefix":"53d1d338090eaa2b","entry":"fetch_json","repo":"lmcache/lmcache","repo_kind":"official","path":"lmcache/cli/http.py","file_url":"https://github.com/lmcache/lmcache/blob/HEAD/lmcache/cli/http.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"53d1d338090eaa2b"}},{"code_sha256_prefix":"aadd3417391aa626","entry":"resolve_flamegraph_dir","repo":"lmcache/lmcache","repo_kind":"official","path":"lmcache/cli/profiling.py","file_url":"https://github.com/lmcache/lmcache/blob/HEAD/lmcache/cli/profiling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aadd3417391aa626"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}