{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/evaluating-quantized-large-language-models","title":"Evaluating Quantized Large Language Models","arxiv_id":"2402.18158","date":"2024-02-28","proceeding":null,"authors":["Shiyao Li","Xuefei Ning","Luning Wang","Tengxuan Liu","Xiangsheng Shi","Shengen Yan","Guohao Dai","Huazhong Yang","Yu Wang"],"abstract":"Post-training quantization (PTQ) has emerged as a promising technique to reduce the cost of large language models (LLMs). Specifically, PTQ can effectively mitigate memory consumption and reduce computational overhead in LLMs. To meet the requirements of both high efficiency and performance across diverse scenarios, a comprehensive evaluation of quantized LLMs is essential to guide the selection of quantization methods. This paper presents a thorough evaluation of these factors by evaluating the effect of PTQ on Weight, Activation, and KV Cache on 11 model families, including OPT, LLaMA2, Falcon, Bloomz, Mistral, ChatGLM, Vicuna, LongChat, StableLM, Gemma, and Mamba, with parameters ranging from 125M to 180B. The evaluation encompasses five types of tasks: basic NLP, emergent ability, trustworthiness, dialogue, and long-context tasks. Moreover, we also evaluate the state-of-the-art (SOTA) quantization methods to demonstrate their applicability. Based on the extensive experiments, we systematically summarize the effect of quantization, provide recommendations to apply quantization techniques, and point out future directions. The code can be found in https://github.com/thu-nics/qllm-eval.","url_abs":"https://arxiv.org/abs/2402.18158v2","url_pdf":"https://arxiv.org/pdf/2402.18158v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"evaluating-quantized-large-language-models","repo_url":"https://github.com/thu-nics/qllm-eval","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"mamba","task_name":"Mamba"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"opt","method_name":"OPT"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.18158","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.18158"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thu-nics/qllm-eval","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"unverified":5},"by_repo_kind":{"official":{"samples":12,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0fbb308a9e0be4b1","entry":"get_act_scale","repo":"thu-nics/qllm-eval","repo_kind":"official","path":"qllm_eval/methods/rep/apply_rep.py","file_url":"https://github.com/thu-nics/qllm-eval/blob/HEAD/qllm_eval/methods/rep/apply_rep.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0fbb308a9e0be4b1"}},{"code_sha256_prefix":"0ae488b77c73c6a6","entry":"get_module_by_name_suffix","repo":"thu-nics/qllm-eval","repo_kind":"official","path":"qllm_eval/utils.py","file_url":"https://github.com/thu-nics/qllm-eval/blob/HEAD/qllm_eval/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0ae488b77c73c6a6"}},{"code_sha256_prefix":"7207245affae2bfe","entry":"get_output_dir","repo":"thu-nics/qllm-eval","repo_kind":"official","path":"qllm_eval/evaluation/q_long/main_longeval.py","file_url":"https://github.com/thu-nics/qllm-eval/blob/HEAD/qllm_eval/evaluation/q_long/main_longeval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7207245affae2bfe"}},{"code_sha256_prefix":"ec909faceaba76f5","entry":"get_weight_scale","repo":"thu-nics/qllm-eval","repo_kind":"official","path":"qllm_eval/methods/rep/apply_rep.py","file_url":"https://github.com/thu-nics/qllm-eval/blob/HEAD/qllm_eval/methods/rep/apply_rep.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec909faceaba76f5"}},{"code_sha256_prefix":"5fdb774f707f7251","entry":"pseudo_quantize_tensor","repo":"thu-nics/qllm-eval","repo_kind":"official","path":"qllm_eval/quantization/quant_funcs.py","file_url":"https://github.com/thu-nics/qllm-eval/blob/HEAD/qllm_eval/quantization/quant_funcs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5fdb774f707f7251"}},{"code_sha256_prefix":"15aceb2649e19457","entry":"quantize_activation_per_token_absmax","repo":"thu-nics/qllm-eval","repo_kind":"official","path":"qllm_eval/quantization/quant_funcs.py","file_url":"https://github.com/thu-nics/qllm-eval/blob/HEAD/qllm_eval/quantization/quant_funcs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"15aceb2649e19457"}},{"code_sha256_prefix":"4a5183ca8bc31006","entry":"quantize_weight_per_channel_absmax","repo":"thu-nics/qllm-eval","repo_kind":"official","path":"qllm_eval/quantization/quant_funcs.py","file_url":"https://github.com/thu-nics/qllm-eval/blob/HEAD/qllm_eval/quantization/quant_funcs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4a5183ca8bc31006"}},{"code_sha256_prefix":"ad238aea2fb2cb76","entry":"append_str_prefix","repo":"thu-nics/qllm-eval","repo_kind":"official","path":"qllm_eval/methods/rep/utils.py","file_url":"https://github.com/thu-nics/qllm-eval/blob/HEAD/qllm_eval/methods/rep/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad238aea2fb2cb76"}},{"code_sha256_prefix":"699b850f943ed723","entry":"build_model_and_enc","repo":"thu-nics/qllm-eval","repo_kind":"official","path":"qllm_eval/utils.py","file_url":"https://github.com/thu-nics/qllm-eval/blob/HEAD/qllm_eval/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"699b850f943ed723"}},{"code_sha256_prefix":"480f04fc99ac2309","entry":"get_op_by_name","repo":"thu-nics/qllm-eval","repo_kind":"official","path":"qllm_eval/methods/rep/utils.py","file_url":"https://github.com/thu-nics/qllm-eval/blob/HEAD/qllm_eval/methods/rep/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"480f04fc99ac2309"}},{"code_sha256_prefix":"10d5993b8024ad65","entry":"get_op_name","repo":"thu-nics/qllm-eval","repo_kind":"official","path":"qllm_eval/methods/rep/utils.py","file_url":"https://github.com/thu-nics/qllm-eval/blob/HEAD/qllm_eval/methods/rep/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"10d5993b8024ad65"}},{"code_sha256_prefix":"57af3572bc6e7d95","entry":"quantize_model","repo":"thu-nics/qllm-eval","repo_kind":"official","path":"qllm_eval/quantization/quant_wrapper.py","file_url":"https://github.com/thu-nics/qllm-eval/blob/HEAD/qllm_eval/quantization/quant_wrapper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"57af3572bc6e7d95"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}