{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/train-small-infer-large-memory-efficient-lora","title":"Train Small, Infer Large: Memory-Efficient LoRA Training for Large Language Models","arxiv_id":"2502.13533","date":"2025-02-19","proceeding":null,"authors":["Jun Zhang","Jue Wang","Huan Li","Lidan Shou","Ke Chen","Yang You","Guiming Xie","Xuejian Gong","Kunlong Zhou"],"abstract":"Large Language Models (LLMs) have significantly advanced natural language processing with exceptional task generalization capabilities. Low-Rank Adaption (LoRA) offers a cost-effective fine-tuning solution, freezing the original model parameters and training only lightweight, low-rank adapter matrices. However, the memory footprint of LoRA is largely dominated by the original model parameters. To mitigate this, we propose LoRAM, a memory-efficient LoRA training scheme founded on the intuition that many neurons in over-parameterized LLMs have low training utility but are essential for inference. LoRAM presents a unique twist: it trains on a pruned (small) model to obtain pruned low-rank matrices, which are then recovered and utilized with the original (large) model for inference. Additionally, minimal-cost continual pre-training, performed by the model publishers in advance, aligns the knowledge discrepancy between pruned and original models. Our extensive experiments demonstrate the efficacy of LoRAM across various pruning strategies and downstream tasks. For a model with 70 billion parameters, LoRAM enables training on a GPU with only 20G HBM, replacing an A100-80G GPU for LoRA training and 15 GPUs for full fine-tuning. Specifically, QLoRAM implemented by structured pruning combined with 4-bit quantization, for LLaMA-3.1-70B (LLaMA-2-70B), reduces the parameter storage cost that dominates the memory usage in low-rank matrix training by 15.81$\\times$ (16.95$\\times$), while achieving dominant performance gains over both the original LLaMA-3.1-70B (LLaMA-2-70B) and LoRA-trained LLaMA-3.1-8B (LLaMA-2-13B).","url_abs":"https://arxiv.org/abs/2502.13533v1","url_pdf":"https://arxiv.org/pdf/2502.13533v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"train-small-infer-large-memory-efficient-lora","repo_url":"https://github.com/junzhang-zj/LoRAM","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":null,"task_name":"GPU"},{"task_slug":"quantization","task_name":"Quantization"}],"methods":[{"method_slug":"adapter","method_name":"Adapter"},{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2502.13533","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.13533"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/junzhang-zj/LoRAM","reach":null}],"summary":{"ran":4,"ran_draft_wrong":1,"unverified":8},"by_repo_kind":{"official":{"samples":12,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"f96dedc8894ebbab","entry":"LlamaMLP","repo":"junzhang-zj/LoRAM","repo_kind":"official","path":"loram/2_pruned_low_rank_matrix_training.py","file_url":"https://github.com/junzhang-zj/LoRAM/blob/HEAD/loram/2_pruned_low_rank_matrix_training.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f96dedc8894ebbab"}},{"code_sha256_prefix":"1ecb31573990e76e","entry":"LlamaPreTrainedModel","repo":"junzhang-zj/LoRAM","repo_kind":"official","path":"loram/2_pruned_low_rank_matrix_training.py","file_url":"https://github.com/junzhang-zj/LoRAM/blob/HEAD/loram/2_pruned_low_rank_matrix_training.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1ecb31573990e76e"}},{"code_sha256_prefix":"9f7bd959c5fde990","entry":"LlamaRMSNorm","repo":"junzhang-zj/LoRAM","repo_kind":"official","path":"loram/2_pruned_low_rank_matrix_training.py","file_url":"https://github.com/junzhang-zj/LoRAM/blob/HEAD/loram/2_pruned_low_rank_matrix_training.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9f7bd959c5fde990"}},{"code_sha256_prefix":"c9f2757f0827b802","entry":"LlamaRotaryEmbedding","repo":"junzhang-zj/LoRAM","repo_kind":"official","path":"loram/2_pruned_low_rank_matrix_training.py","file_url":"https://github.com/junzhang-zj/LoRAM/blob/HEAD/loram/2_pruned_low_rank_matrix_training.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c9f2757f0827b802"}},{"code_sha256_prefix":"583539efd6fd01fb","entry":"apply_rotary_pos_emb","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"583539efd6fd01fb"}},{"code_sha256_prefix":"9dee9df76b1fdd21","entry":"LlamaAttention","repo":"junzhang-zj/LoRAM","repo_kind":"official","path":"loram/2_pruned_low_rank_matrix_training.py","file_url":"https://github.com/junzhang-zj/LoRAM/blob/HEAD/loram/2_pruned_low_rank_matrix_training.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9dee9df76b1fdd21"}},{"code_sha256_prefix":"d4a037311f926e1a","entry":"LlamaDecoderLayer","repo":"junzhang-zj/LoRAM","repo_kind":"official","path":"loram/2_pruned_low_rank_matrix_training.py","file_url":"https://github.com/junzhang-zj/LoRAM/blob/HEAD/loram/2_pruned_low_rank_matrix_training.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d4a037311f926e1a"}},{"code_sha256_prefix":"f2e37ec2f6d5b640","entry":"LlamaFlashAttention2","repo":"junzhang-zj/LoRAM","repo_kind":"official","path":"loram/2_pruned_low_rank_matrix_training.py","file_url":"https://github.com/junzhang-zj/LoRAM/blob/HEAD/loram/2_pruned_low_rank_matrix_training.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f2e37ec2f6d5b640"}},{"code_sha256_prefix":"23f836d2c7d15352","entry":"LlamaForCausalLM","repo":"junzhang-zj/LoRAM","repo_kind":"official","path":"loram/2_pruned_low_rank_matrix_training.py","file_url":"https://github.com/junzhang-zj/LoRAM/blob/HEAD/loram/2_pruned_low_rank_matrix_training.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"23f836d2c7d15352"}},{"code_sha256_prefix":"85f06bba96f8ecb0","entry":"LlamaModel","repo":"junzhang-zj/LoRAM","repo_kind":"official","path":"loram/2_pruned_low_rank_matrix_training.py","file_url":"https://github.com/junzhang-zj/LoRAM/blob/HEAD/loram/2_pruned_low_rank_matrix_training.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"85f06bba96f8ecb0"}},{"code_sha256_prefix":"275b5715a473b63a","entry":"LlamaSdpaAttention","repo":"junzhang-zj/LoRAM","repo_kind":"official","path":"loram/2_pruned_low_rank_matrix_training.py","file_url":"https://github.com/junzhang-zj/LoRAM/blob/HEAD/loram/2_pruned_low_rank_matrix_training.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"275b5715a473b63a"}},{"code_sha256_prefix":"1382890bcd6d3bc9","entry":"main","repo":"junzhang-zj/LoRAM","repo_kind":"official","path":"loram/2_pruned_low_rank_matrix_training.py","file_url":"https://github.com/junzhang-zj/LoRAM/blob/HEAD/loram/2_pruned_low_rank_matrix_training.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1382890bcd6d3bc9"}},{"code_sha256_prefix":"55ec6efd6c53c278","entry":"print_trainable_parameters","repo":"junzhang-zj/LoRAM","repo_kind":"official","path":"loram/2_pruned_low_rank_matrix_training.py","file_url":"https://github.com/junzhang-zj/LoRAM/blob/HEAD/loram/2_pruned_low_rank_matrix_training.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"55ec6efd6c53c278"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}