{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/q-galore-quantized-galore-with-int4","title":"Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients","arxiv_id":"2407.08296","date":"2024-07-11","proceeding":null,"authors":["Zhenyu Zhang","Ajay Jaiswal","Lu Yin","Shiwei Liu","Jiawei Zhao","Yuandong Tian","Zhangyang Wang"],"abstract":"Training Large Language Models (LLMs) is memory-intensive due to the large number of parameters and associated optimization states. GaLore, a recent method, reduces memory usage by projecting weight gradients into a low-rank subspace without compromising performance. However, GaLore relies on time-consuming Singular Value Decomposition (SVD) operations to identify the subspace, and the frequent subspace updates lead to significant training time overhead. Moreover, GaLore offers minimal improvements in accuracy and efficiency compared to LoRA in more accessible fine-tuning scenarios. To address these limitations, we introduce Q-Galore, a novel approach that substantially reduces memory usage by combining quantization and low-rank projection, surpassing the benefits of GaLore. Our method is based on two key observations: (i) the gradient subspace exhibits diverse properties, with some layers converging early in training while others are subject to frequent changes; (ii) the projection matrices are highly resilient to low-bit quantization. Leveraging these insights, Q-GaLore adaptively updates the gradient subspace based on its convergence statistics, achieving comparable performance while significantly reducing the number of SVD operations. We maintain the projection matrices in INT4 format and weights in INT8 format, incorporating stochastic rounding to capture accumulated gradient information. This approach enables a high-precision training trajectory using only low-precision weights. We demonstrate that Q-GaLore achieves highly competitive performance with exceptional memory efficiency. At pre-training, Q-GaLore facilitates training a LLaMA-7B model from scratch on a single NVIDIA RTX 4060 Ti with only 16 GB memory. At fine-tuning, it reduces memory consumption by up to 50% compared to LoRA and GaLore, while consistently outperforming QLoRA at the same memory cost.","url_abs":"https://arxiv.org/abs/2407.08296v1","url_pdf":"https://arxiv.org/pdf/2407.08296v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"q-galore-quantized-galore-with-int4","repo_url":"https://github.com/VITA-Group/Q-GaLore","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"q-galore-quantized-galore-with-int4","repo_url":"https://github.com/jiaweizzhao/galore","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"quantization","task_name":"Quantization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2407.08296","atlas_url":"https://app.syntology.ai/?focus=2407.08296","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.08296"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jiaweizzhao/galore","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/VITA-Group/Q-GaLore","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":1,"ran_draft_wrong":3,"ran":2,"unverified":3},"by_repo_kind":{"official":{"samples":8,"ran":6,"repositories":1},"listed":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f725bc2d76076485","entry":"apply_rotary_pos_emb","repo":"VITA-Group/Q-GaLore","repo_kind":"official","path":"peft_pretraining/modeling_llama.py","file_url":"https://github.com/VITA-Group/Q-GaLore/blob/HEAD/peft_pretraining/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f725bc2d76076485"}},{"code_sha256_prefix":"9abbd152ba303087","entry":"get_cosine_schedule_with_multiple_warmups","repo":"VITA-Group/Q-GaLore","repo_kind":"official","path":"peft_pretraining/training_utils.py","file_url":"https://github.com/VITA-Group/Q-GaLore/blob/HEAD/peft_pretraining/training_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9abbd152ba303087"}},{"code_sha256_prefix":"f58a7fbb020fe7cd","entry":"get_cyclical_cosine_schedule_with_min_lr","repo":"VITA-Group/Q-GaLore","repo_kind":"official","path":"peft_pretraining/training_utils.py","file_url":"https://github.com/VITA-Group/Q-GaLore/blob/HEAD/peft_pretraining/training_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f58a7fbb020fe7cd"}},{"code_sha256_prefix":"b144ce96c9b962b6","entry":"get_scheculer","repo":"VITA-Group/Q-GaLore","repo_kind":"official","path":"peft_pretraining/training_utils.py","file_url":"https://github.com/VITA-Group/Q-GaLore/blob/HEAD/peft_pretraining/training_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b144ce96c9b962b6"}},{"code_sha256_prefix":"bcaa69bdc8b5f8ea","entry":"prepare_model_for_int8_training_simulation","repo":"VITA-Group/Q-GaLore","repo_kind":"official","path":"q_galore_torch/utils/simulate_quantization.py","file_url":"https://github.com/VITA-Group/Q-GaLore/blob/HEAD/q_galore_torch/utils/simulate_quantization.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bcaa69bdc8b5f8ea"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"VITA-Group/Q-GaLore","repo_kind":"official","path":"peft_pretraining/modeling_llama.py","file_url":"https://github.com/VITA-Group/Q-GaLore/blob/HEAD/peft_pretraining/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"c233d4777fc58003","entry":"check_args_torchrun_main","repo":"jiaweizzhao/galore","repo_kind":"listed","path":"peft_pretraining/args_utils.py","file_url":"https://github.com/jiaweizzhao/galore/blob/HEAD/peft_pretraining/args_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c233d4777fc58003"}},{"code_sha256_prefix":"8a11e25793ae28fa","entry":"check_args_torchrun_main","repo":"VITA-Group/Q-GaLore","repo_kind":"official","path":"peft_pretraining/args_utils.py","file_url":"https://github.com/VITA-Group/Q-GaLore/blob/HEAD/peft_pretraining/args_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8a11e25793ae28fa"}},{"code_sha256_prefix":"9cff17b36caed0e9","entry":"prepare_model_for_int8_training","repo":"VITA-Group/Q-GaLore","repo_kind":"official","path":"q_galore_torch/utils/quantization.py","file_url":"https://github.com/VITA-Group/Q-GaLore/blob/HEAD/q_galore_torch/utils/quantization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9cff17b36caed0e9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}