{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/evaluating-the-generalization-ability-of-1","title":"Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolbox","arxiv_id":"2406.12928","date":"2024-06-15","proceeding":null,"authors":["Yijun Liu","Yuan Meng","Fang Wu","Shenhao Peng","Hang Yao","Chaoyu Guan","Chen Tang","Xinzhu Ma","Zhi Wang","Wenwu Zhu"],"abstract":"Large language models (LLMs) have exhibited exciting progress in multiple scenarios, while the huge computational demands hinder their deployments in lots of real-world applications. As an effective means to reduce memory footprint and inference cost, quantization also faces challenges in performance degradation at low bit-widths. Understanding the impact of quantization on LLM capabilities, especially the generalization ability, is crucial. However, the community's main focus remains on the algorithms and models of quantization, with insufficient attention given to whether the quantized models can retain the strong generalization abilities of LLMs. In this work, we fill this gap by providing a comprehensive benchmark suite for this research topic, including an evaluation system, detailed analyses, and a general toolbox. Specifically, based on the dominant pipeline in LLM quantization, we primarily explore the impact of calibration data distribution on the generalization of quantized LLMs and conduct the benchmark using more than 40 datasets within two main scenarios. Based on this benchmark, we conduct extensive experiments with two well-known LLMs (English and Chinese) and four quantization algorithms to investigate this topic in-depth, yielding several counter-intuitive and valuable findings, e.g., models quantized using a calibration set with the same distribution as the test data are not necessarily optimal. Besides, to facilitate future research, we also release a modular-designed toolbox, which decouples the overall pipeline into several separate components, e.g., base LLM module, dataset module, quantizer module, etc. and allows subsequent researchers to easily assemble their methods through a simple configuration. Our benchmark suite is publicly available at https://github.com/TsingmaoAI/MI-optimize","url_abs":"https://arxiv.org/abs/2406.12928v1","url_pdf":"https://arxiv.org/pdf/2406.12928v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"evaluating-the-generalization-ability-of-1","repo_url":"https://github.com/tsingmaoai/mi-optimize","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"base","method_name":"BASE"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.12928","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.12928"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tsingmaoai/mi-optimize","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":4,"unverified":8},"by_repo_kind":{"official":{"samples":12,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"d89391706db541b9","entry":"get_boss","repo":"tsingmaoai/mi-optimize","repo_kind":"official","path":"mi_optimize/datasets/load_boss.py","file_url":"https://github.com/tsingmaoai/mi-optimize/blob/HEAD/mi_optimize/datasets/load_boss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d89391706db541b9"}},{"code_sha256_prefix":"40419b9c9281dcf9","entry":"get_subjects_ceval","repo":"tsingmaoai/mi-optimize","repo_kind":"official","path":"mi_optimize/datasets/load_ceval.py","file_url":"https://github.com/tsingmaoai/mi-optimize/blob/HEAD/mi_optimize/datasets/load_ceval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"40419b9c9281dcf9"}},{"code_sha256_prefix":"91b9c29f9f68f094","entry":"get_subjects_cmmlu","repo":"tsingmaoai/mi-optimize","repo_kind":"official","path":"mi_optimize/datasets/load_cmmlu.py","file_url":"https://github.com/tsingmaoai/mi-optimize/blob/HEAD/mi_optimize/datasets/load_cmmlu.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"91b9c29f9f68f094"}},{"code_sha256_prefix":"cf57d949ad2d6f45","entry":"load_cmmlu","repo":"tsingmaoai/mi-optimize","repo_kind":"official","path":"mi_optimize/datasets/load_cmmlu.py","file_url":"https://github.com/tsingmaoai/mi-optimize/blob/HEAD/mi_optimize/datasets/load_cmmlu.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"cf57d949ad2d6f45"}},{"code_sha256_prefix":"0f169d61e5b5e02a","entry":"get_c4","repo":"tsingmaoai/mi-optimize","repo_kind":"official","path":"mi_optimize/datasets/data_loader.py","file_url":"https://github.com/tsingmaoai/mi-optimize/blob/HEAD/mi_optimize/datasets/data_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0f169d61e5b5e02a"}},{"code_sha256_prefix":"b1df7bc50114300b","entry":"get_calibrate_boss","repo":"tsingmaoai/mi-optimize","repo_kind":"official","path":"mi_optimize/datasets/load_boss.py","file_url":"https://github.com/tsingmaoai/mi-optimize/blob/HEAD/mi_optimize/datasets/load_boss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b1df7bc50114300b"}},{"code_sha256_prefix":"d71510096bae581e","entry":"get_ceval","repo":"tsingmaoai/mi-optimize","repo_kind":"official","path":"mi_optimize/datasets/load_ceval.py","file_url":"https://github.com/tsingmaoai/mi-optimize/blob/HEAD/mi_optimize/datasets/load_ceval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d71510096bae581e"}},{"code_sha256_prefix":"1909079fb191a012","entry":"get_cmmlu","repo":"tsingmaoai/mi-optimize","repo_kind":"official","path":"mi_optimize/datasets/load_cmmlu.py","file_url":"https://github.com/tsingmaoai/mi-optimize/blob/HEAD/mi_optimize/datasets/load_cmmlu.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1909079fb191a012"}},{"code_sha256_prefix":"f6e95ffd6622a1b4","entry":"get_ptb","repo":"tsingmaoai/mi-optimize","repo_kind":"official","path":"mi_optimize/datasets/data_loader.py","file_url":"https://github.com/tsingmaoai/mi-optimize/blob/HEAD/mi_optimize/datasets/data_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f6e95ffd6622a1b4"}},{"code_sha256_prefix":"fc292f3230d4dfd8","entry":"get_str","repo":"tsingmaoai/mi-optimize","repo_kind":"official","path":"mi_optimize/datasets/load_boss.py","file_url":"https://github.com/tsingmaoai/mi-optimize/blob/HEAD/mi_optimize/datasets/load_boss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"fc292f3230d4dfd8"}},{"code_sha256_prefix":"05cb56deff8da528","entry":"get_wikitext2","repo":"tsingmaoai/mi-optimize","repo_kind":"official","path":"mi_optimize/datasets/data_loader.py","file_url":"https://github.com/tsingmaoai/mi-optimize/blob/HEAD/mi_optimize/datasets/data_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"05cb56deff8da528"}},{"code_sha256_prefix":"10664b9521178405","entry":"load_ceval","repo":"tsingmaoai/mi-optimize","repo_kind":"official","path":"mi_optimize/datasets/load_ceval.py","file_url":"https://github.com/tsingmaoai/mi-optimize/blob/HEAD/mi_optimize/datasets/load_ceval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"10664b9521178405"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}