{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/excp-extreme-llm-checkpoint-compression-via","title":"ExCP: Extreme LLM Checkpoint Compression via Weight-Momentum Joint Shrinking","arxiv_id":"2406.11257","date":"2024-06-17","proceeding":null,"authors":["Wenshuo Li","Xinghao Chen","Han Shu","Yehui Tang","Yunhe Wang"],"abstract":"Large language models (LLM) have recently attracted significant attention in the field of artificial intelligence. However, the training process of these models poses significant challenges in terms of computational and storage capacities, thus compressing checkpoints has become an urgent problem. In this paper, we propose a novel Extreme Checkpoint Compression (ExCP) framework, which significantly reduces the required storage of training checkpoints while achieving nearly lossless performance. We first calculate the residuals of adjacent checkpoints to obtain the essential but sparse information for higher compression ratio. To further excavate the redundancy parameters in checkpoints, we then propose a weight-momentum joint shrinking method to utilize another important information during the model optimization, i.e., momentum. In particular, we exploit the information of both model and optimizer to discard as many parameters as possible while preserving critical information to ensure optimal performance. Furthermore, we utilize non-uniform quantization to further compress the storage of checkpoints. We extensively evaluate our proposed ExCP framework on several models ranging from 410M to 7B parameters and demonstrate significant storage reduction while maintaining strong performance. For instance, we achieve approximately $70\\times$ compression for the Pythia-410M model, with the final performance being as accurate as the original model on various downstream tasks. Codes will be available at https://github.com/Gaffey/ExCP.","url_abs":"https://arxiv.org/abs/2406.11257v1","url_pdf":"https://arxiv.org/pdf/2406.11257v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"excp-extreme-llm-checkpoint-compression-via","repo_url":"https://github.com/gaffey/excp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"model-optimization","task_name":"Model Optimization"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.11257","atlas_url":"https://app.syntology.ai/?focus=2406.11257","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.11257"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Gaffey/ExCP","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gaffey/excp","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":2,"ran_honours":1,"ran":6,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":10,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8a38d7c525fad8f1","entry":"accuracy","repo":"gaffey/excp","repo_kind":"official","path":"LMtrainer/pretrain.py","file_url":"https://github.com/gaffey/excp/blob/HEAD/LMtrainer/pretrain.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8a38d7c525fad8f1"}},{"code_sha256_prefix":"33de97c702d46371","entry":"divide","repo":"gaffey/excp","repo_kind":"official","path":"LMtrainer/mpu/utils.py","file_url":"https://github.com/gaffey/excp/blob/HEAD/LMtrainer/mpu/utils.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"33de97c702d46371"}},{"code_sha256_prefix":"c12b8cc9acc7e10f","entry":"fault_tolerance_data_collator","repo":"gaffey/excp","repo_kind":"official","path":"LMtrainer/pretrain.py","file_url":"https://github.com/gaffey/excp/blob/HEAD/LMtrainer/pretrain.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c12b8cc9acc7e10f"}},{"code_sha256_prefix":"c2727b034c1d4655","entry":"get_pythia_config","repo":"gaffey/excp","repo_kind":"official","path":"LMtrainer/pretrain.py","file_url":"https://github.com/gaffey/excp/blob/HEAD/LMtrainer/pretrain.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c2727b034c1d4655"}},{"code_sha256_prefix":"9d020e96607ddadd","entry":"prune","repo":"Gaffey/ExCP","repo_kind":"official","path":"compress_pythia.py","file_url":"https://github.com/Gaffey/ExCP/blob/HEAD/compress_pythia.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9d020e96607ddadd"}},{"code_sha256_prefix":"8358aaebb9f596b7","entry":"prune_optimizer","repo":"Gaffey/ExCP","repo_kind":"official","path":"compress_pythia.py","file_url":"https://github.com/Gaffey/ExCP/blob/HEAD/compress_pythia.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8358aaebb9f596b7"}},{"code_sha256_prefix":"5e61758e06c2d266","entry":"quant","repo":"gaffey/excp","repo_kind":"official","path":"compress_pythia.py","file_url":"https://github.com/gaffey/excp/blob/HEAD/compress_pythia.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5e61758e06c2d266"}},{"code_sha256_prefix":"ff0731adadbb3e8f","entry":"split_tensor_along_last_dim","repo":"gaffey/excp","repo_kind":"official","path":"LMtrainer/mpu/utils.py","file_url":"https://github.com/gaffey/excp/blob/HEAD/LMtrainer/mpu/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ff0731adadbb3e8f"}},{"code_sha256_prefix":"ada1a7cf46c42f0c","entry":"traverse_cache_root","repo":"gaffey/excp","repo_kind":"official","path":"LMtrainer/data_utils/traverse_utils.py","file_url":"https://github.com/gaffey/excp/blob/HEAD/LMtrainer/data_utils/traverse_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ada1a7cf46c42f0c"}},{"code_sha256_prefix":"1b1aa00e64679073","entry":"travese_json_root","repo":"gaffey/excp","repo_kind":"official","path":"LMtrainer/data_utils/traverse_utils.py","file_url":"https://github.com/gaffey/excp/blob/HEAD/LMtrainer/data_utils/traverse_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1b1aa00e64679073"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}