{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/llmcbench-benchmarking-large-language-model","title":"LLMCBench: Benchmarking Large Language Model Compression for Efficient Deployment","arxiv_id":"2410.21352","date":"2024-10-28","proceeding":null,"authors":["Ge Yang","Changyi He","Jinyang Guo","Jianyu Wu","Yifu Ding","Aishan Liu","Haotong Qin","Pengliang Ji","Xianglong Liu"],"abstract":"Although large language models (LLMs) have demonstrated their strong intelligence ability, the high demand for computation and storage hinders their practical application. To this end, many model compression techniques are proposed to increase the efficiency of LLMs. However, current researches only validate their methods on limited models, datasets, metrics, etc, and still lack a comprehensive evaluation under more general scenarios. So it is still a question of which model compression approach we should use under a specific case. To mitigate this gap, we present the Large Language Model Compression Benchmark (LLMCBench), a rigorously designed benchmark with an in-depth analysis for LLM compression algorithms. We first analyze the actual model production requirements and carefully design evaluation tracks and metrics. Then, we conduct extensive experiments and comparison using multiple mainstream LLM compression approaches. Finally, we perform an in-depth analysis based on the evaluation and provide useful insight for LLM compression design. We hope our LLMCBench can contribute insightful suggestions for LLM compression algorithm design and serve as a foundation for future research. Our code is available at https://github.com/AboveParadise/LLMCBench.","url_abs":"https://arxiv.org/abs/2410.21352v2","url_pdf":"https://arxiv.org/pdf/2410.21352v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"llmcbench-benchmarking-large-language-model","repo_url":"https://github.com/aboveparadise/llmcbench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"model-compression","task_name":"Model Compression"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.21352","atlas_url":"https://app.syntology.ai/?focus=2410.21352","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.21352"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/AboveParadise/LLMCBench","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aboveparadise/llmcbench","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":3,"ran":1,"unverified":13},"by_repo_kind":{"official":{"samples":17,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"cd763eaf1ac287e7","entry":"format_example","repo":"AboveParadise/LLMCBench","repo_kind":"official","path":"evaluate_mmlu.py","file_url":"https://github.com/AboveParadise/LLMCBench/blob/HEAD/evaluate_mmlu.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cd763eaf1ac287e7"}},{"code_sha256_prefix":"18c6154c3dcf6170","entry":"format_frame","repo":"AboveParadise/LLMCBench","repo_kind":"official","path":"evaluate_tQA.py","file_url":"https://github.com/AboveParadise/LLMCBench/blob/HEAD/evaluate_tQA.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"18c6154c3dcf6170"}},{"code_sha256_prefix":"6ab745408cb8648b","entry":"format_subject","repo":"AboveParadise/LLMCBench","repo_kind":"official","path":"evaluate_mmlu.py","file_url":"https://github.com/AboveParadise/LLMCBench/blob/HEAD/evaluate_mmlu.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6ab745408cb8648b"}},{"code_sha256_prefix":"3383488ea82b6c15","entry":"gen_prompt","repo":"AboveParadise/LLMCBench","repo_kind":"official","path":"evaluate_mmlu.py","file_url":"https://github.com/AboveParadise/LLMCBench/blob/HEAD/evaluate_mmlu.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3383488ea82b6c15"}},{"code_sha256_prefix":"5a00707831004560","entry":"eval","repo":"AboveParadise/LLMCBench","repo_kind":"official","path":"evaluate_mnli.py","file_url":"https://github.com/AboveParadise/LLMCBench/blob/HEAD/evaluate_mnli.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a00707831004560"}},{"code_sha256_prefix":"d8fea7ed2cb92813","entry":"eval_ppl","repo":"AboveParadise/LLMCBench","repo_kind":"official","path":"evaluate_ppl.py","file_url":"https://github.com/AboveParadise/LLMCBench/blob/HEAD/evaluate_ppl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d8fea7ed2cb92813"}},{"code_sha256_prefix":"dd3a73b173924da9","entry":"find_subsequence","repo":"AboveParadise/LLMCBench","repo_kind":"official","path":"evaluate_tQA.py","file_url":"https://github.com/AboveParadise/LLMCBench/blob/HEAD/evaluate_tQA.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dd3a73b173924da9"}},{"code_sha256_prefix":"158b8847881a64b3","entry":"format_best","repo":"AboveParadise/LLMCBench","repo_kind":"official","path":"evaluate_tQA.py","file_url":"https://github.com/AboveParadise/LLMCBench/blob/HEAD/evaluate_tQA.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"158b8847881a64b3"}},{"code_sha256_prefix":"984d3567b5126f65","entry":"format_example","repo":"AboveParadise/LLMCBench","repo_kind":"official","path":"evaluate_advglue.py","file_url":"https://github.com/AboveParadise/LLMCBench/blob/HEAD/evaluate_advglue.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"984d3567b5126f65"}},{"code_sha256_prefix":"7538653cf2565390","entry":"format_example","repo":"AboveParadise/LLMCBench","repo_kind":"official","path":"evaluate_mnli.py","file_url":"https://github.com/AboveParadise/LLMCBench/blob/HEAD/evaluate_mnli.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7538653cf2565390"}},{"code_sha256_prefix":"dafa37632f3f91bf","entry":"format_example","repo":"AboveParadise/LLMCBench","repo_kind":"official","path":"evaluate_qnli.py","file_url":"https://github.com/AboveParadise/LLMCBench/blob/HEAD/evaluate_qnli.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dafa37632f3f91bf"}},{"code_sha256_prefix":"00e220ac67451517","entry":"gen_prompt","repo":"AboveParadise/LLMCBench","repo_kind":"official","path":"evaluate_advglue.py","file_url":"https://github.com/AboveParadise/LLMCBench/blob/HEAD/evaluate_advglue.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"00e220ac67451517"}},{"code_sha256_prefix":"b8a3329c6d548881","entry":"gen_prompt","repo":"AboveParadise/LLMCBench","repo_kind":"official","path":"evaluate_mnli.py","file_url":"https://github.com/AboveParadise/LLMCBench/blob/HEAD/evaluate_mnli.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b8a3329c6d548881"}},{"code_sha256_prefix":"65cc2c5bebc786da","entry":"gen_prompt","repo":"AboveParadise/LLMCBench","repo_kind":"official","path":"evaluate_qnli.py","file_url":"https://github.com/AboveParadise/LLMCBench/blob/HEAD/evaluate_qnli.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"65cc2c5bebc786da"}},{"code_sha256_prefix":"3f1d2bce76741bc0","entry":"get_loaders","repo":"AboveParadise/LLMCBench","repo_kind":"official","path":"evaluate_ppl.py","file_url":"https://github.com/AboveParadise/LLMCBench/blob/HEAD/evaluate_ppl.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3f1d2bce76741bc0"}},{"code_sha256_prefix":"2e1841f87a930398","entry":"load_model_tokenizer","repo":"AboveParadise/LLMCBench","repo_kind":"official","path":"evaluate_advglue.py","file_url":"https://github.com/AboveParadise/LLMCBench/blob/HEAD/evaluate_advglue.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2e1841f87a930398"}},{"code_sha256_prefix":"a58462e59e01f858","entry":"load_questions","repo":"AboveParadise/LLMCBench","repo_kind":"official","path":"tqa_utilities.py","file_url":"https://github.com/AboveParadise/LLMCBench/blob/HEAD/tqa_utilities.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a58462e59e01f858"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}