{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/parameter-efficient-multi-task-fine-tuning","title":"Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks","arxiv_id":"2106.04489","date":"2021-06-08","proceeding":"ACL 2021 5","authors":["Rabeeh Karimi Mahabadi","Sebastian Ruder","Mostafa Dehghani","James Henderson"],"abstract":"State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model. However, such modules are trained separately for each task and thus do not enable sharing information across tasks. In this paper, we show that we can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks, which condition on task, adapter position, and layer id in a transformer model. This parameter-efficient multi-task learning framework allows us to achieve the best of both worlds by sharing knowledge across tasks via hypernetworks while enabling the model to adapt to each individual task through task-specific adapters. Experiments on the well-known GLUE benchmark show improved performance in multi-task learning while adding only 0.29% parameters per task. We additionally demonstrate substantial performance improvements in few-shot domain generalization across a variety of tasks. Our code is publicly available in https://github.com/rabeehk/hyperformer.","url_abs":"https://arxiv.org/abs/2106.04489v1","url_pdf":"https://arxiv.org/pdf/2106.04489v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"parameter-efficient-multi-task-fine-tuning","repo_url":"https://github.com/rabeehk/hyperformer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"domain-generalization","task_name":"Domain Generalization"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"multi-task-learning","task_name":"Multi-Task Learning"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"}],"methods":[{"method_slug":"adapter","method_name":"Adapter"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.04489","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.04489"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rabeehk/hyperformer","reach":null}],"summary":{"ran":3,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c4d48dab1da65576","entry":"AdapterHyperNet","repo":"rabeehk/hyperformer","repo_kind":"official","path":"hyperformer/adapters/adapter_controller.py","file_url":"https://github.com/rabeehk/hyperformer/blob/HEAD/hyperformer/adapters/adapter_controller.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c4d48dab1da65576"}},{"code_sha256_prefix":"7929f8a733e5bfbc","entry":"LayerNormHyperNet","repo":"rabeehk/hyperformer","repo_kind":"official","path":"hyperformer/adapters/adapter_controller.py","file_url":"https://github.com/rabeehk/hyperformer/blob/HEAD/hyperformer/adapters/adapter_controller.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7929f8a733e5bfbc"}},{"code_sha256_prefix":"7e8ce70c56f6a890","entry":"MetaAdapterController","repo":"rabeehk/hyperformer","repo_kind":"official","path":"hyperformer/adapters/adapter_controller.py","file_url":"https://github.com/rabeehk/hyperformer/blob/HEAD/hyperformer/adapters/adapter_controller.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7e8ce70c56f6a890"}},{"code_sha256_prefix":"2abb955708c84118","entry":"linear_layer","repo":"rabeehk/hyperformer","repo_kind":"official","path":"hyperformer/adapters/adapter_controller.py","file_url":"https://github.com/rabeehk/hyperformer/blob/HEAD/hyperformer/adapters/adapter_controller.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2abb955708c84118"}},{"code_sha256_prefix":"d4d0162fd3f6583b","entry":"init_linear_layer","repo":"rabeehk/hyperformer","repo_kind":"official","path":"hyperformer/adapters/adapter_controller.py","file_url":"https://github.com/rabeehk/hyperformer/blob/HEAD/hyperformer/adapters/adapter_controller.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d4d0162fd3f6583b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}