{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-a-unified-view-of-parameter-efficient-1","title":"Towards a Unified View of Parameter-Efficient Transfer Learning","arxiv_id":"2110.04366","date":"2021-10-08","proceeding":"ICLR 2022 4","authors":["Junxian He","Chunting Zhou","Xuezhe Ma","Taylor Berg-Kirkpatrick","Graham Neubig"],"abstract":"Fine-tuning large pre-trained language models on downstream tasks has become the de-facto learning paradigm in NLP. However, conventional approaches fine-tune all the parameters of the pre-trained model, which becomes prohibitive as the model size and the number of tasks grow. Recent work has proposed a variety of parameter-efficient transfer learning methods that only fine-tune a small number of (extra) parameters to attain strong performance. While effective, the critical ingredients for success and the connections among the various methods are poorly understood. In this paper, we break down the design of state-of-the-art parameter-efficient transfer learning methods and present a unified framework that establishes connections between them. Specifically, we re-frame them as modifications to specific hidden states in pre-trained models, and define a set of design dimensions along which different methods vary, such as the function to compute the modification and the position to apply the modification. Through comprehensive empirical studies across machine translation, text summarization, language understanding, and text classification benchmarks, we utilize the unified view to identify important design choices in previous methods. Furthermore, our unified framework enables the transfer of design elements across different approaches, and as a result we are able to instantiate new parameter-efficient fine-tuning methods that tune less parameters than previous methods while being more effective, achieving comparable results to fine-tuning all parameters on all four tasks.","url_abs":"https://arxiv.org/abs/2110.04366v3","url_pdf":"https://arxiv.org/pdf/2110.04366v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-a-unified-view-of-parameter-efficient-1","repo_url":"https://github.com/jxhe/unify-parameter-efficient-tuning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"text-classification","task_name":"Text Classification"},{"task_slug":"text-summarization","task_name":"Text Summarization"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"},{"task_slug":"text-classification-1","task_name":"text-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2110.04366","atlas_url":"https://app.syntology.ai/?focus=2110.04366","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.04366"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jxhe/unify-parameter-efficient-tuning","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":8,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":11,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"e758ef19d32b8b11","entry":"adapter_func","repo":"jxhe/unify-parameter-efficient-tuning","repo_kind":"official","path":"petl/petl_factory.py","file_url":"https://github.com/jxhe/unify-parameter-efficient-tuning/blob/HEAD/petl/petl_factory.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"e758ef19d32b8b11"}},{"code_sha256_prefix":"7fca652b013370e4","entry":"compute_params","repo":"jxhe/unify-parameter-efficient-tuning","repo_kind":"official","path":"analysis/paper_plot.py","file_url":"https://github.com/jxhe/unify-parameter-efficient-tuning/blob/HEAD/analysis/paper_plot.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"7fca652b013370e4"}},{"code_sha256_prefix":"2a7cfa36f9e48900","entry":"format","repo":"jxhe/unify-parameter-efficient-tuning","repo_kind":"official","path":"analysis/paper_plot.py","file_url":"https://github.com/jxhe/unify-parameter-efficient-tuning/blob/HEAD/analysis/paper_plot.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"2a7cfa36f9e48900"}},{"code_sha256_prefix":"a4475703ff58ecf9","entry":"gelu_fast","repo":"jxhe/unify-parameter-efficient-tuning","repo_kind":"official","path":"src/transformers/activations.py","file_url":"https://github.com/jxhe/unify-parameter-efficient-tuning/blob/HEAD/src/transformers/activations.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"a4475703ff58ecf9"}},{"code_sha256_prefix":"45bb87451230d5e8","entry":"gelu_new","repo":"jxhe/unify-parameter-efficient-tuning","repo_kind":"official","path":"src/transformers/activations.py","file_url":"https://github.com/jxhe/unify-parameter-efficient-tuning/blob/HEAD/src/transformers/activations.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"45bb87451230d5e8"}},{"code_sha256_prefix":"e21f72f1a9834aef","entry":"quick_gelu","repo":"jxhe/unify-parameter-efficient-tuning","repo_kind":"official","path":"src/transformers/activations.py","file_url":"https://github.com/jxhe/unify-parameter-efficient-tuning/blob/HEAD/src/transformers/activations.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"e21f72f1a9834aef"}},{"code_sha256_prefix":"a6d9f5f488eb99a9","entry":"safe_round","repo":"jxhe/unify-parameter-efficient-tuning","repo_kind":"official","path":"petl/utils.py","file_url":"https://github.com/jxhe/unify-parameter-efficient-tuning/blob/HEAD/petl/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"a6d9f5f488eb99a9"}},{"code_sha256_prefix":"a75e6ade7e6f9240","entry":"softmax_gating","repo":"jxhe/unify-parameter-efficient-tuning","repo_kind":"official","path":"petl/petl_factory.py","file_url":"https://github.com/jxhe/unify-parameter-efficient-tuning/blob/HEAD/petl/petl_factory.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"a75e6ade7e6f9240"}},{"code_sha256_prefix":"88b0f6942509de64","entry":"sortish_sampler_indices","repo":"jxhe/unify-parameter-efficient-tuning","repo_kind":"official","path":"petl/dynamic_batching.py","file_url":"https://github.com/jxhe/unify-parameter-efficient-tuning/blob/HEAD/petl/dynamic_batching.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"88b0f6942509de64"}},{"code_sha256_prefix":"37a5eed2dbd663ca","entry":"gelu_fast","repo":"jxhe/unify-parameter-efficient-tuning","repo_kind":"official","path":"src/transformers/activations_tf.py","file_url":"https://github.com/jxhe/unify-parameter-efficient-tuning/blob/HEAD/src/transformers/activations_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"37a5eed2dbd663ca"}},{"code_sha256_prefix":"cc8c8c3ebf0c343f","entry":"mish","repo":"jxhe/unify-parameter-efficient-tuning","repo_kind":"official","path":"src/transformers/activations_tf.py","file_url":"https://github.com/jxhe/unify-parameter-efficient-tuning/blob/HEAD/src/transformers/activations_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"cc8c8c3ebf0c343f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}