{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reducing-fine-tuning-memory-overhead-by","title":"Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation","arxiv_id":"2406.16282","date":"2024-06-24","proceeding":null,"authors":["Yuchen Yang","Yingdong Shi","Cheems Wang","XianTong Zhen","Yuxuan Shi","Jun Xu"],"abstract":"Fine-tuning pretrained large models to downstream tasks is an important problem, which however suffers from huge memory overhead due to large-scale parameters. This work strives to reduce memory overhead in fine-tuning from perspectives of activation function and layer normalization. To this end, we propose the Approximate Backpropagation (Approx-BP) theory, which provides the theoretical feasibility of decoupling the forward and backward passes. We apply our Approx-BP theory to backpropagation training and derive memory-efficient alternatives of GELU and SiLU activation functions, which use derivative functions of ReLUs in the backward pass while keeping their forward pass unchanged. In addition, we introduce a Memory-Sharing Backpropagation strategy, which enables the activation memory to be shared by two adjacent layers, thereby removing activation memory usage redundancy. Our method neither induces extra computation nor reduces training efficiency. We conduct extensive experiments with pretrained vision and language models, and the results demonstrate that our proposal can reduce up to $\\sim$$30\\%$ of the peak memory usage. Our code is released at https://github.com/yyyyychen/LowMemoryBP.","url_abs":"https://arxiv.org/abs/2406.16282v1","url_pdf":"https://arxiv.org/pdf/2406.16282v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reducing-fine-tuning-memory-overhead-by","repo_url":"https://github.com/yyyyychen/LowMemoryBP","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"silu","method_name":"SiLU"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.16282","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.16282"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yyyyychen/LowMemoryBP","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":8,"ran_honours":2,"unverified":11},"by_repo_kind":{"official":{"samples":21,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"454817035cd7e70d","entry":"acc_and_f1","repo":"yyyyychen/LowMemoryBP","repo_kind":"official","path":"experiments/roberta_exp/metric.py","file_url":"https://github.com/yyyyychen/LowMemoryBP/blob/HEAD/experiments/roberta_exp/metric.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"454817035cd7e70d"}},{"code_sha256_prefix":"23d97b310f421cf4","entry":"dgelu","repo":"yyyyychen/lowmemorybp","repo_kind":"official","path":"search_act/sa_search.py","file_url":"https://github.com/yyyyychen/lowmemorybp/blob/HEAD/search_act/sa_search.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"23d97b310f421cf4"}},{"code_sha256_prefix":"5bab00057953eed3","entry":"dgelu","repo":"yyyyychen/LowMemoryBP","repo_kind":"official","path":"search_act/plot_results.py","file_url":"https://github.com/yyyyychen/LowMemoryBP/blob/HEAD/search_act/plot_results.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5bab00057953eed3"}},{"code_sha256_prefix":"d166f5c139047d8f","entry":"dgelu","repo":"yyyyychen/LowMemoryBP","repo_kind":"official","path":"search_act/sa_search.py","file_url":"https://github.com/yyyyychen/LowMemoryBP/blob/HEAD/search_act/sa_search.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d166f5c139047d8f"}},{"code_sha256_prefix":"7fcedfc60ac103d5","entry":"gelu","repo":"yyyyychen/lowmemorybp","repo_kind":"official","path":"search_act/sa_search.py","file_url":"https://github.com/yyyyychen/lowmemorybp/blob/HEAD/search_act/sa_search.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7fcedfc60ac103d5"}},{"code_sha256_prefix":"92d332f9f6d8d916","entry":"gelu","repo":"yyyyychen/LowMemoryBP","repo_kind":"official","path":"search_act/plot_results.py","file_url":"https://github.com/yyyyychen/LowMemoryBP/blob/HEAD/search_act/plot_results.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"92d332f9f6d8d916"}},{"code_sha256_prefix":"bd4ce1c1da8ac7dd","entry":"get_gelu_truncate_bounds","repo":"yyyyychen/LowMemoryBP","repo_kind":"official","path":"search_act/sgd_search.py","file_url":"https://github.com/yyyyychen/LowMemoryBP/blob/HEAD/search_act/sgd_search.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd4ce1c1da8ac7dd"}},{"code_sha256_prefix":"9852f5e220563896","entry":"get_silu_truncate_bounds","repo":"yyyyychen/LowMemoryBP","repo_kind":"official","path":"search_act/sgd_search.py","file_url":"https://github.com/yyyyychen/LowMemoryBP/blob/HEAD/search_act/sgd_search.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9852f5e220563896"}},{"code_sha256_prefix":"e3e13638a674e4b1","entry":"pearson_and_spearman","repo":"yyyyychen/LowMemoryBP","repo_kind":"official","path":"experiments/roberta_exp/metric.py","file_url":"https://github.com/yyyyychen/LowMemoryBP/blob/HEAD/experiments/roberta_exp/metric.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e3e13638a674e4b1"}},{"code_sha256_prefix":"df56b7b4df2b7d62","entry":"silu","repo":"yyyyychen/LowMemoryBP","repo_kind":"official","path":"search_act/plot_results.py","file_url":"https://github.com/yyyyychen/LowMemoryBP/blob/HEAD/search_act/plot_results.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"df56b7b4df2b7d62"}},{"code_sha256_prefix":"55935d62b8500881","entry":"gelu","repo":"yyyyychen/LowMemoryBP","repo_kind":"official","path":"search_act/sa_search.py","file_url":"https://github.com/yyyyychen/LowMemoryBP/blob/HEAD/search_act/sa_search.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"55935d62b8500881"}},{"code_sha256_prefix":"24f25e03d2510fc4","entry":"layer_norm","repo":"yyyyychen/LowMemoryBP","repo_kind":"official","path":"lomem/normalization.py","file_url":"https://github.com/yyyyychen/LowMemoryBP/blob/HEAD/lomem/normalization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"24f25e03d2510fc4"}},{"code_sha256_prefix":"2e7e3714bf4747c5","entry":"pack_bool_to_uint8","repo":"yyyyychen/LowMemoryBP","repo_kind":"official","path":"lomem/packing.py","file_url":"https://github.com/yyyyychen/LowMemoryBP/blob/HEAD/lomem/packing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2e7e3714bf4747c5"}},{"code_sha256_prefix":"aa093de845df408e","entry":"regelu2","repo":"yyyyychen/LowMemoryBP","repo_kind":"official","path":"lomem/activation.py","file_url":"https://github.com/yyyyychen/LowMemoryBP/blob/HEAD/lomem/activation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa093de845df408e"}},{"code_sha256_prefix":"1eafd3b53f875c65","entry":"replace_linear4bit","repo":"yyyyychen/LowMemoryBP","repo_kind":"official","path":"experiments/llama_exp/qlora.py","file_url":"https://github.com/yyyyychen/LowMemoryBP/blob/HEAD/experiments/llama_exp/qlora.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1eafd3b53f875c65"}},{"code_sha256_prefix":"91254aa095e9a8e4","entry":"resilu2","repo":"yyyyychen/LowMemoryBP","repo_kind":"official","path":"lomem/activation.py","file_url":"https://github.com/yyyyychen/LowMemoryBP/blob/HEAD/lomem/activation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"91254aa095e9a8e4"}},{"code_sha256_prefix":"dd3123886d069aa0","entry":"rms_norm","repo":"yyyyychen/LowMemoryBP","repo_kind":"official","path":"lomem/normalization.py","file_url":"https://github.com/yyyyychen/LowMemoryBP/blob/HEAD/lomem/normalization.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dd3123886d069aa0"}},{"code_sha256_prefix":"fc28c3f7ab3434fd","entry":"silu","repo":"yyyyychen/lowmemorybp","repo_kind":"official","path":"search_act/sa_search.py","file_url":"https://github.com/yyyyychen/lowmemorybp/blob/HEAD/search_act/sa_search.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fc28c3f7ab3434fd"}},{"code_sha256_prefix":"fbb6daa55e3cf79c","entry":"silu","repo":"yyyyychen/LowMemoryBP","repo_kind":"official","path":"search_act/sa_search.py","file_url":"https://github.com/yyyyychen/LowMemoryBP/blob/HEAD/search_act/sa_search.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fbb6daa55e3cf79c"}},{"code_sha256_prefix":"83eba990d34db6a2","entry":"simple_accuracy","repo":"yyyyychen/LowMemoryBP","repo_kind":"official","path":"experiments/roberta_exp/metric.py","file_url":"https://github.com/yyyyychen/LowMemoryBP/blob/HEAD/experiments/roberta_exp/metric.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"83eba990d34db6a2"}},{"code_sha256_prefix":"2d38c0e161fa803b","entry":"unpack_uint8_to_bool","repo":"yyyyychen/LowMemoryBP","repo_kind":"official","path":"lomem/packing.py","file_url":"https://github.com/yyyyychen/LowMemoryBP/blob/HEAD/lomem/packing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2d38c0e161fa803b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}