{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lst-ladder-side-tuning-for-parameter-and","title":"LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning","arxiv_id":"2206.06522","date":"2022-06-13","proceeding":null,"authors":["Yi-Lin Sung","Jaemin Cho","Mohit Bansal"],"abstract":"Fine-tuning large pre-trained models on downstream tasks has been adopted in a variety of domains recently. However, it is costly to update the entire parameter set of large pre-trained models. Although recently proposed parameter-efficient transfer learning (PETL) techniques allow updating a small subset of parameters (e.g. only using 2% of parameters) inside a pre-trained backbone network for a new task, they only reduce the training memory requirement by up to 30%. This is because the gradient computation for the trainable parameters still requires backpropagation through the large pre-trained backbone model. To address this, we propose Ladder Side-Tuning (LST), a new PETL technique that can reduce training memory requirements by more substantial amounts. Unlike existing parameter-efficient methods that insert additional parameters inside backbone networks, we train a ladder side network, a small and separate network that takes intermediate activations as input via shortcut connections (called ladders) from backbone networks and makes predictions. LST has significantly lower memory requirements than previous methods, because it does not require backpropagation through the backbone network, but instead only through the side network and ladder connections. We evaluate our method with various models (T5 and CLIP-T5) on both NLP (GLUE) and vision-and-language (VQA, GQA, NLVR2 , MSCOCO) tasks. LST saves 69% of the memory costs to fine-tune the whole network, while other methods only save 26% of that in similar parameter usages (hence, 2.7x more memory savings). Moreover, LST achieves higher accuracy than Adapter and LoRA in a low-memory regime. To further show the advantage of this better memory efficiency, we also apply LST to larger T5 models, attaining better GLUE performance than full fine-tuning and other PETL methods. The accuracy-efficiency trade-off also holds on VL tasks.","url_abs":"https://arxiv.org/abs/2206.06522v2","url_pdf":"https://arxiv.org/pdf/2206.06522v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lst-ladder-side-tuning-for-parameter-and","repo_url":"https://github.com/ylsung/ladder-side-tuning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"lst-ladder-side-tuning-for-parameter-and","repo_url":"https://github.com/zirui-ray-liu/wtacrs","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}}],"tasks":[{"task_slug":"transfer-learning","task_name":"Transfer Learning"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[{"method_slug":"adafactor","method_name":"Adafactor"},{"method_slug":"adapter","method_name":"Adapter"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"glu","method_name":"Gated Linear Unit"},{"method_slug":"inverse-square-root-schedule","method_name":"Inverse Square Root Schedule"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"sentencepiece","method_name":"SentencePiece"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"t5","method_name":"T5"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2206.06522","atlas_url":"https://app.syntology.ai/?focus=2206.06522","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.06522"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ylsung/ladder-side-tuning","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zirui-ray-liu/wtacrs","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":3},"by_repo_kind":{"official":{"samples":6,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e21bb85b8e1b4836","entry":"do_nms","repo":"ylsung/ladder-side-tuning","repo_kind":"official","path":"VL-T5/inference/modeling_frcnn.py","file_url":"https://github.com/ylsung/ladder-side-tuning/blob/HEAD/VL-T5/inference/modeling_frcnn.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e21bb85b8e1b4836"}},{"code_sha256_prefix":"cfa4887da544c990","entry":"norm_box","repo":"ylsung/ladder-side-tuning","repo_kind":"official","path":"VL-T5/inference/modeling_frcnn.py","file_url":"https://github.com/ylsung/ladder-side-tuning/blob/HEAD/VL-T5/inference/modeling_frcnn.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cfa4887da544c990"}},{"code_sha256_prefix":"54a2fff601881b32","entry":"pad_list_tensors","repo":"ylsung/ladder-side-tuning","repo_kind":"official","path":"VL-T5/inference/modeling_frcnn.py","file_url":"https://github.com/ylsung/ladder-side-tuning/blob/HEAD/VL-T5/inference/modeling_frcnn.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"54a2fff601881b32"}},{"code_sha256_prefix":"f3a241789ee26538","entry":"build_model","repo":"ylsung/ladder-side-tuning","repo_kind":"official","path":"CLIP-ViL/clip/model.py","file_url":"https://github.com/ylsung/ladder-side-tuning/blob/HEAD/CLIP-ViL/clip/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f3a241789ee26538"}},{"code_sha256_prefix":"9826533300048f5e","entry":"get_optimizer","repo":"ylsung/ladder-side-tuning","repo_kind":"official","path":"CLIP-ViL/src/param.py","file_url":"https://github.com/ylsung/ladder-side-tuning/blob/HEAD/CLIP-ViL/src/param.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9826533300048f5e"}},{"code_sha256_prefix":"6011ad1e831dede2","entry":"load_obj_tsv","repo":"ylsung/ladder-side-tuning","repo_kind":"official","path":"CLIP-ViL/src/utils.py","file_url":"https://github.com/ylsung/ladder-side-tuning/blob/HEAD/CLIP-ViL/src/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6011ad1e831dede2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}