{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/4-bit-shampoo-for-memory-efficient-network","title":"4-bit Shampoo for Memory-Efficient Network Training","arxiv_id":"2405.18144","date":"2024-05-28","proceeding":null,"authors":["Sike Wang","Pan Zhou","Jia Li","Hua Huang"],"abstract":"Second-order optimizers, maintaining a matrix termed a preconditioner, are superior to first-order optimizers in both theory and practice. The states forming the preconditioner and its inverse root restrict the maximum size of models trained by second-order optimizers. To address this, compressing 32-bit optimizer states to lower bitwidths has shown promise in reducing memory usage. However, current approaches only pertain to first-order optimizers. In this paper, we propose the first 4-bit second-order optimizers, exemplified by 4-bit Shampoo, maintaining performance similar to that of 32-bit ones. We show that quantizing the eigenvector matrix of the preconditioner in 4-bit Shampoo is remarkably better than quantizing the preconditioner itself both theoretically and experimentally. By rectifying the orthogonality of the quantized eigenvector matrix, we enhance the approximation of the preconditioner's eigenvector matrix, which also benefits the computation of its inverse 4-th root. Besides, we find that linear square quantization slightly outperforms dynamic tree quantization when quantizing second-order optimizer states. Evaluation on various networks for image classification and natural language modeling demonstrates that our 4-bit Shampoo achieves comparable performance to its 32-bit counterpart while being more memory-efficient.","url_abs":"https://arxiv.org/abs/2405.18144v3","url_pdf":"https://arxiv.org/pdf/2405.18144v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"4-bit-shampoo-for-memory-efficient-network","repo_url":"https://github.com/sike-wang/low-bit-shampoo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"quantization","task_name":"Quantization"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.18144","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.18144"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sike-wang/low-bit-shampoo","reach":{"status":"ok"}}],"summary":{"ran":13,"unverified":2},"by_repo_kind":{"official":{"samples":15,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"7f680e86e6397b23","entry":"MatPower","repo":"sike-wang/low-bit-shampoo","repo_kind":"official","path":"optimizers/matrix_functions.py","file_url":"https://github.com/sike-wang/low-bit-shampoo/blob/HEAD/optimizers/matrix_functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7f680e86e6397b23"}},{"code_sha256_prefix":"567029bd46065708","entry":"densenet121","repo":"sike-wang/low-bit-shampoo","repo_kind":"official","path":"models/densenet.py","file_url":"https://github.com/sike-wang/low-bit-shampoo/blob/HEAD/models/densenet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"567029bd46065708"}},{"code_sha256_prefix":"3ed7580937fb0008","entry":"densenet169","repo":"sike-wang/low-bit-shampoo","repo_kind":"official","path":"models/densenet.py","file_url":"https://github.com/sike-wang/low-bit-shampoo/blob/HEAD/models/densenet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3ed7580937fb0008"}},{"code_sha256_prefix":"b70932e817d72f6c","entry":"densenet201","repo":"sike-wang/low-bit-shampoo","repo_kind":"official","path":"models/densenet.py","file_url":"https://github.com/sike-wang/low-bit-shampoo/blob/HEAD/models/densenet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b70932e817d72f6c"}},{"code_sha256_prefix":"71ae4764866d8ea8","entry":"resnet18","repo":"sike-wang/low-bit-shampoo","repo_kind":"official","path":"models/resnet.py","file_url":"https://github.com/sike-wang/low-bit-shampoo/blob/HEAD/models/resnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"71ae4764866d8ea8"}},{"code_sha256_prefix":"8422d19ea2f1ab19","entry":"resnet34","repo":"sike-wang/low-bit-shampoo","repo_kind":"official","path":"models/resnet.py","file_url":"https://github.com/sike-wang/low-bit-shampoo/blob/HEAD/models/resnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8422d19ea2f1ab19"}},{"code_sha256_prefix":"aa756ebc41f95062","entry":"resnet50","repo":"sike-wang/low-bit-shampoo","repo_kind":"official","path":"models/resnet.py","file_url":"https://github.com/sike-wang/low-bit-shampoo/blob/HEAD/models/resnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa756ebc41f95062"}},{"code_sha256_prefix":"1b9cf937758df695","entry":"swin_tiny","repo":"sike-wang/low-bit-shampoo","repo_kind":"official","path":"models/swin.py","file_url":"https://github.com/sike-wang/low-bit-shampoo/blob/HEAD/models/swin.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1b9cf937758df695"}},{"code_sha256_prefix":"05abe9f6c5347322","entry":"vgg11","repo":"sike-wang/low-bit-shampoo","repo_kind":"official","path":"models/vgg.py","file_url":"https://github.com/sike-wang/low-bit-shampoo/blob/HEAD/models/vgg.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"05abe9f6c5347322"}},{"code_sha256_prefix":"d77d79a6e5db71b8","entry":"vgg13","repo":"sike-wang/low-bit-shampoo","repo_kind":"official","path":"models/vgg.py","file_url":"https://github.com/sike-wang/low-bit-shampoo/blob/HEAD/models/vgg.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d77d79a6e5db71b8"}},{"code_sha256_prefix":"0579755ab192dafc","entry":"vgg16","repo":"sike-wang/low-bit-shampoo","repo_kind":"official","path":"models/vgg.py","file_url":"https://github.com/sike-wang/low-bit-shampoo/blob/HEAD/models/vgg.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0579755ab192dafc"}},{"code_sha256_prefix":"6ecf696df373b05d","entry":"vit_base_32_224","repo":"sike-wang/low-bit-shampoo","repo_kind":"official","path":"models/vit.py","file_url":"https://github.com/sike-wang/low-bit-shampoo/blob/HEAD/models/vit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6ecf696df373b05d"}},{"code_sha256_prefix":"42ddec61c2f4cb83","entry":"vit_small","repo":"sike-wang/low-bit-shampoo","repo_kind":"official","path":"models/vit.py","file_url":"https://github.com/sike-wang/low-bit-shampoo/blob/HEAD/models/vit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"42ddec61c2f4cb83"}},{"code_sha256_prefix":"d77abe9d00f96a18","entry":"ComputePower","repo":"sike-wang/low-bit-shampoo","repo_kind":"official","path":"optimizers/matrix_functions.py","file_url":"https://github.com/sike-wang/low-bit-shampoo/blob/HEAD/optimizers/matrix_functions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d77abe9d00f96a18"}},{"code_sha256_prefix":"3e19268a6eb951a3","entry":"PowerIter","repo":"sike-wang/low-bit-shampoo","repo_kind":"official","path":"optimizers/matrix_functions.py","file_url":"https://github.com/sike-wang/low-bit-shampoo/blob/HEAD/optimizers/matrix_functions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3e19268a6eb951a3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}