{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/llm-inference-unveiled-survey-and-roofline","title":"LLM Inference Unveiled: Survey and Roofline Model Insights","arxiv_id":"2402.16363","date":"2024-02-26","proceeding":null,"authors":["Zhihang Yuan","Yuzhang Shang","Yang Zhou","Zhen Dong","Zhe Zhou","Chenhao Xue","Bingzhe Wu","Zhikai Li","Qingyi Gu","Yong Jae Lee","Yan Yan","Beidi Chen","Guangyu Sun","Kurt Keutzer"],"abstract":"The field of efficient Large Language Model (LLM) inference is rapidly evolving, presenting a unique blend of opportunities and challenges. Although the field has expanded and is vibrant, there hasn't been a concise framework that analyzes the various methods of LLM Inference to provide a clear understanding of this domain. Our survey stands out from traditional literature reviews by not only summarizing the current state of research but also by introducing a framework based on roofline model for systematic analysis of LLM inference techniques. This framework identifies the bottlenecks when deploying LLMs on hardware devices and provides a clear understanding of practical problems, such as why LLMs are memory-bound, how much memory and computation they need, and how to choose the right hardware. We systematically collate the latest advancements in efficient LLM inference, covering crucial areas such as model compression (e.g., Knowledge Distillation and Quantization), algorithm improvements (e.g., Early Exit and Mixture-of-Expert), and both hardware and system-level enhancements. Our survey stands out by analyzing these methods with roofline model, helping us understand their impact on memory access and computation. This distinctive approach not only showcases the current research landscape but also delivers valuable insights for practical implementation, positioning our work as an indispensable resource for researchers new to the field as well as for those seeking to deepen their understanding of efficient LLM deployment. The analyze tool, LLM-Viewer, is open-sourced.","url_abs":"https://arxiv.org/abs/2402.16363v6","url_pdf":"https://arxiv.org/pdf/2402.16363v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"llm-inference-unveiled-survey-and-roofline","repo_url":"https://github.com/hahnyuan/llm-viewer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"llm-inference-unveiled-survey-and-roofline","repo_url":"https://github.com/hahnyuan/llmviewer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"model-compression","task_name":"Model Compression"},{"task_slug":"quantization","task_name":"Quantization"},{"task_slug":"survey","task_name":"Survey"},{"task_slug":"model","task_name":"model"}],"methods":[{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.16363","atlas_url":"https://app.syntology.ai/?focus=2402.16363","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.16363"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hahnyuan/llmviewer","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hahnyuan/llm-viewer","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":10},"by_repo_kind":{"official":{"samples":10,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"53674017e8412a02","entry":"get_hidden_size","repo":"hahnyuan/llm-viewer","repo_kind":"official","path":"configs/DiT.py","file_url":"https://github.com/hahnyuan/llm-viewer/blob/HEAD/configs/DiT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"53674017e8412a02"}},{"code_sha256_prefix":"c317a05d4ea52fdb","entry":"get_num_attention_heads","repo":"hahnyuan/llm-viewer","repo_kind":"official","path":"configs/DiT.py","file_url":"https://github.com/hahnyuan/llm-viewer/blob/HEAD/configs/DiT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c317a05d4ea52fdb"}},{"code_sha256_prefix":"71a53e6b26a90a3b","entry":"get_num_attention_heads","repo":"hahnyuan/llm-viewer","repo_kind":"official","path":"configs/Llama.py","file_url":"https://github.com/hahnyuan/llm-viewer/blob/HEAD/configs/Llama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"71a53e6b26a90a3b"}},{"code_sha256_prefix":"0974dc3884b2024c","entry":"get_num_key_value_heads","repo":"hahnyuan/llm-viewer","repo_kind":"official","path":"configs/DiT.py","file_url":"https://github.com/hahnyuan/llm-viewer/blob/HEAD/configs/DiT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0974dc3884b2024c"}},{"code_sha256_prefix":"d8114ca5ce58674d","entry":"get_num_key_value_heads","repo":"hahnyuan/llm-viewer","repo_kind":"official","path":"configs/Llama.py","file_url":"https://github.com/hahnyuan/llm-viewer/blob/HEAD/configs/Llama.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d8114ca5ce58674d"}},{"code_sha256_prefix":"5e0ebeceb00f510c","entry":"get_num_key_value_heads","repo":"hahnyuan/llm-viewer","repo_kind":"official","path":"configs/chatglm3.py","file_url":"https://github.com/hahnyuan/llm-viewer/blob/HEAD/configs/chatglm3.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e0ebeceb00f510c"}},{"code_sha256_prefix":"f8884ef464c9898f","entry":"get_quant_bit","repo":"hahnyuan/llm-viewer","repo_kind":"official","path":"get_model_graph.py","file_url":"https://github.com/hahnyuan/llm-viewer/blob/HEAD/get_model_graph.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f8884ef464c9898f"}},{"code_sha256_prefix":"fa31a6cb9a3df793","entry":"roofline_analyze","repo":"hahnyuan/llm-viewer","repo_kind":"official","path":"roofline_model.py","file_url":"https://github.com/hahnyuan/llm-viewer/blob/HEAD/roofline_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fa31a6cb9a3df793"}},{"code_sha256_prefix":"a16e1a0be1f021fa","entry":"str_number","repo":"hahnyuan/llm-viewer","repo_kind":"official","path":"utils.py","file_url":"https://github.com/hahnyuan/llm-viewer/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a16e1a0be1f021fa"}},{"code_sha256_prefix":"31eb55b64fcedb77","entry":"str_number_time","repo":"hahnyuan/llm-viewer","repo_kind":"official","path":"utils.py","file_url":"https://github.com/hahnyuan/llm-viewer/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"31eb55b64fcedb77"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}