{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/apiserve-efficient-api-support-for-large","title":"InferCept: Efficient Intercept Support for Augmented Large Language Model Inference","arxiv_id":"2402.01869","date":"2024-02-02","proceeding":null,"authors":["Reyna Abhyankar","Zijian He","Vikranth Srivatsa","Hao Zhang","Yiying Zhang"],"abstract":"Large language models are increasingly integrated with external environments, tools, and agents like ChatGPT plugins to extend their capability beyond language-centric tasks. However, today's LLM inference systems are designed for standalone LLMs. They treat each external interaction as the end of LLM generation and form a new request when the interaction finishes, causing unnecessary recomputation of already computed contexts, which accounts for 37-40% of total model forwarding time. This paper presents InferCept, the first LLM inference framework targeting augmented LLMs and supporting the efficient interception of LLM generation. InferCept minimizes the GPU resource waste caused by LLM interceptions and dedicates saved memory for serving more requests. InferCept improves the overall serving throughput by 1.6x-2x and completes 2x more requests per second compared to the state-of-the-art LLM inference systems.","url_abs":"https://arxiv.org/abs/2402.01869v2","url_pdf":"https://arxiv.org/pdf/2402.01869v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"apiserve-efficient-api-support-for-large","repo_url":"https://github.com/wuklab/infercept","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.01869","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.01869"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wuklab/infercept","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"84a26133838eab27","entry":"sample_requests","repo":"wuklab/infercept","repo_kind":"official","path":"benchmarks/benchmark_api_serving.py","file_url":"https://github.com/wuklab/infercept/blob/HEAD/benchmarks/benchmark_api_serving.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"84a26133838eab27"}},{"code_sha256_prefix":"f4e08ad9325905dc","entry":"sample_requests","repo":"wuklab/infercept","repo_kind":"official","path":"benchmarks/benchmark_serving.py","file_url":"https://github.com/wuklab/infercept/blob/HEAD/benchmarks/benchmark_serving.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f4e08ad9325905dc"}},{"code_sha256_prefix":"ed11039cd60ea03f","entry":"sample_requests","repo":"wuklab/infercept","repo_kind":"official","path":"benchmarks/benchmark_throughput.py","file_url":"https://github.com/wuklab/infercept/blob/HEAD/benchmarks/benchmark_throughput.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ed11039cd60ea03f"}},{"code_sha256_prefix":"62f75e14e50e58ad","entry":"run_hf","repo":"wuklab/infercept","repo_kind":"official","path":"benchmarks/benchmark_throughput.py","file_url":"https://github.com/wuklab/infercept/blob/HEAD/benchmarks/benchmark_throughput.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"62f75e14e50e58ad"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}