{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2604-06664","title":"Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Start","arxiv_id":"2604.06664","date":"2026-04-08","proceeding":null,"authors":["Xueshen Liu","Yongji Wu","Yuncheng Yao","Danyang Zhuo","Ion Stoica","Z. Morley Mao"],"abstract":"Modern LLM service providers increasingly rely on autoscaling and parallelism reconfiguration to respond to rapidly changing workloads, but cold-start latency remains a major bottleneck. While recent systems have reduced model weight loading to seconds, CUDA graph capture still takes tens of seconds to minutes and often dominates startup. Unfortunately, CUDA graphs cannot be naively serialized: beyond graph topology, they are tightly coupled to execution context, including device addresses embedded in kernel arguments and kernel code lazily loaded during warmup. Existing approaches either rely on brittle kernel-specific patching or heavyweight process-level checkpoint/restore that are inflexible to dynamic parallelism switching. We present Foundry, a template-based CUDA graph context materialization system that persists both graph topology and execution context during an offline processing stage, and reconstructs executable graphs online with negligible overhead. Foundry enforces deterministic memory layouts, automatically extracts and reloads kernel binaries required by captured graphs, and reduces online reconstruction costs through topology-based templating. For distributed serving, Foundry further enables a single-GPU offline capture to generate templates for multi-GPU deployments by patching only rank-dependent communication state. Across dense and MoE models up to 235B parameters, Foundry reduces cold-start latency by up to 99%, cutting the initialization time of Qwen3-235B-A22B from 10 minutes to 3.9 seconds while preserving the throughput gains of CUDA graphs.","url_abs":"https://arxiv.org/abs/2604.06664","url_pdf":"https://arxiv.org/pdf/2604.06664","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2604.06664","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2604.06664"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/foundry-org/foundry","reach":null}],"summary":{"unverified":4},"by_repo_kind":{"found_in_text":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5223c02622af1660","entry":"compute_workspace_rank","repo":"foundry-org/foundry","repo_kind":"found_in_text","path":"python/foundry/integration/sglang/config.py","file_url":"https://github.com/foundry-org/foundry/blob/HEAD/python/foundry/integration/sglang/config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5223c02622af1660"}},{"code_sha256_prefix":"054a5b6e5c2dc78e","entry":"create_warmup_state","repo":"foundry-org/foundry","repo_kind":"found_in_text","path":"python/foundry/integration/sglang/runtime.py","file_url":"https://github.com/foundry-org/foundry/blob/HEAD/python/foundry/integration/sglang/runtime.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"054a5b6e5c2dc78e"}},{"code_sha256_prefix":"f52ddbe79c811c18","entry":"load_warmup_state","repo":"foundry-org/foundry","repo_kind":"found_in_text","path":"python/foundry/integration/vllm/runtime.py","file_url":"https://github.com/foundry-org/foundry/blob/HEAD/python/foundry/integration/vllm/runtime.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f52ddbe79c811c18"}},{"code_sha256_prefix":"57b76bf34c34c3fe","entry":"parse_size","repo":"foundry-org/foundry","repo_kind":"found_in_text","path":"python/foundry/allocation_region.py","file_url":"https://github.com/foundry-org/foundry/blob/HEAD/python/foundry/allocation_region.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"57b76bf34c34c3fe"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}