{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/is-c4-dataset-optimal-for-pruning-an","title":"Is C4 Dataset Optimal for Pruning? An Investigation of Calibration Data for LLM Pruning","arxiv_id":"2410.07461","date":"2024-10-09","proceeding":null,"authors":["Abhinav Bandari","Lu Yin","Cheng-Yu Hsieh","Ajay Kumar Jaiswal","Tianlong Chen","Li Shen","Ranjay Krishna","Shiwei Liu"],"abstract":"Network pruning has emerged as a potential solution to make LLMs cheaper to deploy. However, existing LLM pruning approaches universally rely on the C4 dataset as the calibration data for calculating pruning scores, leaving its optimality unexplored. In this study, we evaluate the choice of calibration data on LLM pruning, across a wide range of datasets that are most commonly used in LLM training and evaluation, including four pertaining datasets as well as three categories of downstream tasks encompassing nine datasets. Each downstream dataset is prompted with In-Context Learning (ICL) and Chain-of-Thought (CoT), respectively. Besides the already intriguing observation that the choice of calibration data significantly impacts the performance of pruned LLMs, our results also uncover several subtle and often unexpected findings, summarized as follows: (1) C4 is not the optimal choice for LLM pruning, even among commonly used pre-training datasets; (2) arithmetic datasets, when used as calibration data, performs on par or even better than pre-training datasets; (3) pruning with downstream datasets does not necessarily help the corresponding downstream task, compared to pre-training data; (4) ICL is widely beneficial to all data categories, whereas CoT is only useful on certain tasks. Our findings shed light on the importance of carefully selecting calibration data for LLM pruning and pave the way for more efficient deployment of these powerful models in real-world applications. We release our code at: https://github.com/abx393/llm-pruning-calibration-data.","url_abs":"https://arxiv.org/abs/2410.07461v1","url_pdf":"https://arxiv.org/pdf/2410.07461v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"is-c4-dataset-optimal-for-pruning-an","repo_url":"https://github.com/abx393/llm-pruning-calibration-data","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"network-pruning","task_name":"Network Pruning"}],"methods":[{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.07461","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.07461"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/abx393/llm-pruning-calibration-data","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4,"ran_draft_wrong":1,"unverified":4},"by_repo_kind":{"official":{"samples":9,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"91a952a7c3e11f8b","entry":"check_sparsity","repo":"abx393/llm-pruning-calibration-data","repo_kind":"official","path":"lib/prune.py","file_url":"https://github.com/abx393/llm-pruning-calibration-data/blob/HEAD/lib/prune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"91a952a7c3e11f8b"}},{"code_sha256_prefix":"2ed16fcdc14ad951","entry":"find_layers","repo":"abx393/llm-pruning-calibration-data","repo_kind":"official","path":"lib/prune.py","file_url":"https://github.com/abx393/llm-pruning-calibration-data/blob/HEAD/lib/prune.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2ed16fcdc14ad951"}},{"code_sha256_prefix":"a7a81b4c3a6ce8aa","entry":"grouper","repo":"abx393/llm-pruning-calibration-data","repo_kind":"official","path":"lib/tasks/util.py","file_url":"https://github.com/abx393/llm-pruning-calibration-data/blob/HEAD/lib/tasks/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a7a81b4c3a6ce8aa"}},{"code_sha256_prefix":"4dae8adb46b1e46e","entry":"prepare_calibration_input","repo":"abx393/llm-pruning-calibration-data","repo_kind":"official","path":"lib/prune.py","file_url":"https://github.com/abx393/llm-pruning-calibration-data/blob/HEAD/lib/prune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4dae8adb46b1e46e"}},{"code_sha256_prefix":"cdb488c7e605e47b","entry":"shrink_seq","repo":"abx393/llm-pruning-calibration-data","repo_kind":"official","path":"lib/tasks/util.py","file_url":"https://github.com/abx393/llm-pruning-calibration-data/blob/HEAD/lib/tasks/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cdb488c7e605e47b"}},{"code_sha256_prefix":"558bf8c612039e2f","entry":"get_ellipses","repo":"abx393/llm-pruning-calibration-data","repo_kind":"official","path":"lib/data.py","file_url":"https://github.com/abx393/llm-pruning-calibration-data/blob/HEAD/lib/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"558bf8c612039e2f"}},{"code_sha256_prefix":"de40864022e3d0d0","entry":"get_random","repo":"abx393/llm-pruning-calibration-data","repo_kind":"official","path":"lib/data.py","file_url":"https://github.com/abx393/llm-pruning-calibration-data/blob/HEAD/lib/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"de40864022e3d0d0"}},{"code_sha256_prefix":"8be36298f96f0805","entry":"get_wikitext2","repo":"abx393/llm-pruning-calibration-data","repo_kind":"official","path":"lib/data.py","file_url":"https://github.com/abx393/llm-pruning-calibration-data/blob/HEAD/lib/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8be36298f96f0805"}},{"code_sha256_prefix":"5d1116e05e5772c7","entry":"process_request","repo":"abx393/llm-pruning-calibration-data","repo_kind":"official","path":"lib/tasks/eval_harness.py","file_url":"https://github.com/abx393/llm-pruning-calibration-data/blob/HEAD/lib/tasks/eval_harness.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5d1116e05e5772c7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}