{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fine-tuning-quantized-neural-networks-with","title":"Fine-tuning Quantized Neural Networks with Zeroth-order Optimization","arxiv_id":"2505.13430","date":"2025-05-19","proceeding":null,"authors":["Sifeng Shang","Jiayi Zhou","Chenyu Lin","Minxian Li","Kaiyang Zhou"],"abstract":"As the size of large language models grows exponentially, GPU memory has become a bottleneck for adapting these models to downstream tasks. In this paper, we aim to push the limits of memory-efficient training by minimizing memory usage on model weights, gradients, and optimizer states, within a unified framework. Our idea is to eliminate both gradients and optimizer states using zeroth-order optimization, which approximates gradients by perturbing weights during forward passes to identify gradient directions. To minimize memory usage on weights, we employ model quantization, e.g., converting from bfloat16 to int4. However, directly applying zeroth-order optimization to quantized weights is infeasible due to the precision gap between discrete weights and continuous gradients, which would otherwise require de-quantization and re-quantization. To overcome this challenge, we propose Quantized Zeroth-order Optimization (QZO), a novel approach that perturbs the continuous quantization scale for gradient estimation and uses a directional derivative clipping method to stabilize training. QZO is orthogonal to both scalar-based and codebook-based post-training quantization methods. Compared to full-parameter fine-tuning in bfloat16, QZO can reduce the total memory cost by more than 18$\\times$ for 4-bit LLMs, and enables fine-tuning Llama-2-13B and Stable Diffusion 3.5 Large within a single 24GB GPU.","url_abs":"https://arxiv.org/abs/2505.13430v1","url_pdf":"https://arxiv.org/pdf/2505.13430v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fine-tuning-quantized-neural-networks-with","repo_url":"https://github.com/maifoundations/qzo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2505.13430","atlas_url":"https://app.syntology.ai/?focus=2505.13430","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.13430"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/maifoundations/qzo","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":8,"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"official":{"samples":15,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e82be1ebda714ec3","entry":"encode_prompt","repo":"maifoundations/qzo","repo_kind":"official","path":"large_language_models/utils.py","file_url":"https://github.com/maifoundations/qzo/blob/HEAD/large_language_models/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e82be1ebda714ec3"}},{"code_sha256_prefix":"93a86bf0447b5071","entry":"encode_sd3_prompt_with_clip","repo":"maifoundations/qzo","repo_kind":"official","path":"stable_diffusion/text_to_image/train_utils.py","file_url":"https://github.com/maifoundations/qzo/blob/HEAD/stable_diffusion/text_to_image/train_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"93a86bf0447b5071"}},{"code_sha256_prefix":"2b6f26327780ae00","entry":"encode_sd3_prompt_with_t5","repo":"maifoundations/qzo","repo_kind":"official","path":"stable_diffusion/text_to_image/train_utils.py","file_url":"https://github.com/maifoundations/qzo/blob/HEAD/stable_diffusion/text_to_image/train_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2b6f26327780ae00"}},{"code_sha256_prefix":"54a6124d697bc2c2","entry":"forward_wrap_with_option_len","repo":"maifoundations/qzo","repo_kind":"official","path":"large_language_models/utils.py","file_url":"https://github.com/maifoundations/qzo/blob/HEAD/large_language_models/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"54a6124d697bc2c2"}},{"code_sha256_prefix":"0e575c7c824b1546","entry":"get_calib_dataset","repo":"maifoundations/qzo","repo_kind":"official","path":"large_language_models/quantization.py","file_url":"https://github.com/maifoundations/qzo/blob/HEAD/large_language_models/quantization.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e575c7c824b1546"}},{"code_sha256_prefix":"d7899d043687bf12","entry":"load_tes","repo":"maifoundations/qzo","repo_kind":"official","path":"stable_diffusion/text_to_image/train_utils.py","file_url":"https://github.com/maifoundations/qzo/blob/HEAD/stable_diffusion/text_to_image/train_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d7899d043687bf12"}},{"code_sha256_prefix":"9782bd07382e951c","entry":"next_power_of_2","repo":"maifoundations/qzo","repo_kind":"official","path":"large_language_models/aqlm_inference_kernel/triton_kernel_dequant.py","file_url":"https://github.com/maifoundations/qzo/blob/HEAD/large_language_models/aqlm_inference_kernel/triton_kernel_dequant.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9782bd07382e951c"}},{"code_sha256_prefix":"e7e75981cb464788","entry":"normalize_answer","repo":"maifoundations/qzo","repo_kind":"official","path":"large_language_models/metrics.py","file_url":"https://github.com/maifoundations/qzo/blob/HEAD/large_language_models/metrics.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7e75981cb464788"}},{"code_sha256_prefix":"c66c0606fd369119","entry":"zo_step","repo":"maifoundations/qzo","repo_kind":"official","path":"stable_diffusion/text_to_image/MeZO_utils.py","file_url":"https://github.com/maifoundations/qzo/blob/HEAD/stable_diffusion/text_to_image/MeZO_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c66c0606fd369119"}},{"code_sha256_prefix":"84c658f2c9252878","entry":"calculate_metric","repo":"maifoundations/qzo","repo_kind":"official","path":"large_language_models/metrics.py","file_url":"https://github.com/maifoundations/qzo/blob/HEAD/large_language_models/metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"84c658f2c9252878"}},{"code_sha256_prefix":"b989d4bce26f77ce","entry":"f1","repo":"maifoundations/qzo","repo_kind":"official","path":"large_language_models/metrics.py","file_url":"https://github.com/maifoundations/qzo/blob/HEAD/large_language_models/metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b989d4bce26f77ce"}},{"code_sha256_prefix":"e5420b8f9ec6a920","entry":"get_backward_pass_kernel","repo":"maifoundations/qzo","repo_kind":"official","path":"large_language_models/aqlm_inference_kernel/kernel_selector.py","file_url":"https://github.com/maifoundations/qzo/blob/HEAD/large_language_models/aqlm_inference_kernel/kernel_selector.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e5420b8f9ec6a920"}},{"code_sha256_prefix":"72f47794c604a79d","entry":"get_forward_pass_kernel","repo":"maifoundations/qzo","repo_kind":"official","path":"large_language_models/aqlm_inference_kernel/kernel_selector.py","file_url":"https://github.com/maifoundations/qzo/blob/HEAD/large_language_models/aqlm_inference_kernel/kernel_selector.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"72f47794c604a79d"}},{"code_sha256_prefix":"f7c384f1e681ea7a","entry":"get_task","repo":"maifoundations/qzo","repo_kind":"official","path":"large_language_models/tasks.py","file_url":"https://github.com/maifoundations/qzo/blob/HEAD/large_language_models/tasks.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f7c384f1e681ea7a"}},{"code_sha256_prefix":"4347ff075572d341","entry":"zo_forward","repo":"maifoundations/qzo","repo_kind":"official","path":"stable_diffusion/text_to_image/MeZO_utils.py","file_url":"https://github.com/maifoundations/qzo/blob/HEAD/stable_diffusion/text_to_image/MeZO_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4347ff075572d341"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}