{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/high-throughput-generative-inference-of-large","title":"FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU","arxiv_id":"2303.06865","date":"2023-03-13","proceeding":null,"authors":["Ying Sheng","Lianmin Zheng","Binhang Yuan","Zhuohan Li","Max Ryabinin","Daniel Y. Fu","Zhiqiang Xie","Beidi Chen","Clark Barrett","Joseph E. Gonzalez","Percy Liang","Christopher Ré","Ion Stoica","Ce Zhang"],"abstract":"The high computational and memory requirements of large language model (LLM) inference make it feasible only with multiple high-end accelerators. Motivated by the emerging demand for latency-insensitive tasks with batched processing, this paper initiates the study of high-throughput LLM inference using limited resources, such as a single commodity GPU. We present FlexGen, a high-throughput generation engine for running LLMs with limited GPU memory. FlexGen can be flexibly configured under various hardware resource constraints by aggregating memory and computation from the GPU, CPU, and disk. By solving a linear programming problem, it searches for efficient patterns to store and access tensors. FlexGen further compresses the weights and the attention cache to 4 bits with negligible accuracy loss. These techniques enable FlexGen to have a larger space of batch size choices and thus significantly increase maximum throughput. As a result, when running OPT-175B on a single 16GB GPU, FlexGen achieves significantly higher throughput compared to state-of-the-art offloading systems, reaching a generation throughput of 1 token/s for the first time with an effective batch size of 144. On the HELM benchmark, FlexGen can benchmark a 30B model with a 16GB GPU on 7 representative sub-scenarios in 21 hours. The code is available at https://github.com/FMInference/FlexGen","url_abs":"https://arxiv.org/abs/2303.06865v2","url_pdf":"https://arxiv.org/pdf/2303.06865v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"high-throughput-generative-inference-of-large","repo_url":"https://github.com/fminference/flexgen","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":null,"task_name":"CPU"},{"task_slug":null,"task_name":"GPU"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2303.06865","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.06865"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fminference/flexgen","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":12},"by_repo_kind":{"official":{"samples":12,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1f5e7e1302edb8b7","entry":"benchmark_func","repo":"fminference/flexgen","repo_kind":"official","path":"flexllmgen/profile_bandwidth.py","file_url":"https://github.com/fminference/flexgen/blob/HEAD/flexllmgen/profile_bandwidth.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1f5e7e1302edb8b7"}},{"code_sha256_prefix":"d456b3368cd9ef2c","entry":"best","repo":"fminference/flexgen","repo_kind":"official","path":"experimental/cost_model.py","file_url":"https://github.com/fminference/flexgen/blob/HEAD/experimental/cost_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d456b3368cd9ef2c"}},{"code_sha256_prefix":"80dde39da1ba3708","entry":"compress","repo":"fminference/flexgen","repo_kind":"official","path":"flexllmgen/compression.py","file_url":"https://github.com/fminference/flexgen/blob/HEAD/flexllmgen/compression.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"80dde39da1ba3708"}},{"code_sha256_prefix":"58fb93e90d4ca515","entry":"cut_indices","repo":"fminference/flexgen","repo_kind":"official","path":"flexllmgen/pytorch_backend.py","file_url":"https://github.com/fminference/flexgen/blob/HEAD/flexllmgen/pytorch_backend.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"58fb93e90d4ca515"}},{"code_sha256_prefix":"a2b79de775636d17","entry":"decompress","repo":"fminference/flexgen","repo_kind":"official","path":"flexllmgen/compression.py","file_url":"https://github.com/fminference/flexgen/blob/HEAD/flexllmgen/compression.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a2b79de775636d17"}},{"code_sha256_prefix":"66b0a9c4c8b9fac3","entry":"get_choice","repo":"fminference/flexgen","repo_kind":"official","path":"flexllmgen/flex_opt.py","file_url":"https://github.com/fminference/flexgen/blob/HEAD/flexllmgen/flex_opt.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"66b0a9c4c8b9fac3"}},{"code_sha256_prefix":"9c97f67de32d17d9","entry":"get_compressed_indices","repo":"fminference/flexgen","repo_kind":"official","path":"flexllmgen/compression.py","file_url":"https://github.com/fminference/flexgen/blob/HEAD/flexllmgen/compression.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9c97f67de32d17d9"}},{"code_sha256_prefix":"aa4fd60bcb4735f7","entry":"get_filename","repo":"fminference/flexgen","repo_kind":"official","path":"flexllmgen/flex_opt.py","file_url":"https://github.com/fminference/flexgen/blob/HEAD/flexllmgen/flex_opt.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aa4fd60bcb4735f7"}},{"code_sha256_prefix":"3ade24c053a86925","entry":"get_opt_config","repo":"fminference/flexgen","repo_kind":"official","path":"flexllmgen/opt_config.py","file_url":"https://github.com/fminference/flexgen/blob/HEAD/flexllmgen/opt_config.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3ade24c053a86925"}},{"code_sha256_prefix":"96b89b9b7d6de29e","entry":"piecewise_linear_func","repo":"fminference/flexgen","repo_kind":"official","path":"flexllmgen/utils.py","file_url":"https://github.com/fminference/flexgen/blob/HEAD/flexllmgen/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"96b89b9b7d6de29e"}},{"code_sha256_prefix":"5bd88c5dd640a26b","entry":"piecewise_linear_func_ret_func","repo":"fminference/flexgen","repo_kind":"official","path":"flexllmgen/utils.py","file_url":"https://github.com/fminference/flexgen/blob/HEAD/flexllmgen/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5bd88c5dd640a26b"}},{"code_sha256_prefix":"41f5cd8d4fcde903","entry":"sample_from_range","repo":"fminference/flexgen","repo_kind":"official","path":"flexllmgen/utils.py","file_url":"https://github.com/fminference/flexgen/blob/HEAD/flexllmgen/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"41f5cd8d4fcde903"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}