{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/autopeft-automatic-configuration-search-for","title":"AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning","arxiv_id":"2301.12132","date":"2023-01-28","proceeding":null,"authors":["Han Zhou","Xingchen Wan","Ivan Vulić","Anna Korhonen"],"abstract":"Large pretrained language models are widely used in downstream NLP tasks via task-specific fine-tuning, but such procedures can be costly. Recently, Parameter-Efficient Fine-Tuning (PEFT) methods have achieved strong task performance while updating much fewer parameters than full model fine-tuning (FFT). However, it is non-trivial to make informed design choices on the PEFT configurations, such as their architecture, the number of tunable parameters, and even the layers in which the PEFT modules are inserted. Consequently, it is highly likely that the current, manually designed configurations are suboptimal in terms of their performance-efficiency trade-off. Inspired by advances in neural architecture search, we propose AutoPEFT for automatic PEFT configuration selection: we first design an expressive configuration search space with multiple representative PEFT modules as building blocks. Using multi-objective Bayesian optimisation in a low-cost setup, we then discover a Pareto-optimal set of configurations with strong performance-cost trade-offs across different numbers of parameters that are also highly transferable across different tasks. Empirically, on GLUE and SuperGLUE tasks, we show that AutoPEFT-discovered configurations significantly outperform existing PEFT methods and are on par or better than FFT without incurring substantial training efficiency costs.","url_abs":"https://arxiv.org/abs/2301.12132v3","url_pdf":"https://arxiv.org/pdf/2301.12132v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"autopeft-automatic-configuration-search-for","repo_url":"https://github.com/cambridgeltl/autopeft","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"bayesian-optimisation","task_name":"Bayesian Optimisation"},{"task_slug":"architecture-search","task_name":"Neural Architecture Search"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2301.12132","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2301.12132"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cambridgeltl/autopeft","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2,"ran_draft_wrong":1,"unverified":10},"by_repo_kind":{"official":{"samples":12,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"454817035cd7e70d","entry":"acc_and_f1","repo":"cambridgeltl/autopeft","repo_kind":"official","path":"glue_metrics.py","file_url":"https://github.com/cambridgeltl/autopeft/blob/HEAD/glue_metrics.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"454817035cd7e70d"}},{"code_sha256_prefix":"54fa343f001c7cff","entry":"kronecker_product","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"54fa343f001c7cff"}},{"code_sha256_prefix":"e3e13638a674e4b1","entry":"pearson_and_spearman","repo":"cambridgeltl/autopeft","repo_kind":"official","path":"glue_metrics.py","file_url":"https://github.com/cambridgeltl/autopeft/blob/HEAD/glue_metrics.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"e3e13638a674e4b1"}},{"code_sha256_prefix":"37a5eed2dbd663ca","entry":"gelu_fast","repo":"cambridgeltl/autopeft","repo_kind":"official","path":"adapter-transformers-adapters3.1.0/src/transformers/activations_tf.py","file_url":"https://github.com/cambridgeltl/autopeft/blob/HEAD/adapter-transformers-adapters3.1.0/src/transformers/activations_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"37a5eed2dbd663ca"}},{"code_sha256_prefix":"280a696268513b28","entry":"get_EI","repo":"cambridgeltl/autopeft","repo_kind":"official","path":"adapterhub/bo/models.py","file_url":"https://github.com/cambridgeltl/autopeft/blob/HEAD/adapterhub/bo/models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"280a696268513b28"}},{"code_sha256_prefix":"2f0f77422d5933be","entry":"get_all_checkpoint","repo":"cambridgeltl/autopeft","repo_kind":"official","path":"adapterhub/nas_search.py","file_url":"https://github.com/cambridgeltl/autopeft/blob/HEAD/adapterhub/nas_search.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"2f0f77422d5933be"}},{"code_sha256_prefix":"8f025929c3e1325b","entry":"get_kernel","repo":"cambridgeltl/autopeft","repo_kind":"official","path":"adapterhub/bo/models.py","file_url":"https://github.com/cambridgeltl/autopeft/blob/HEAD/adapterhub/bo/models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"8f025929c3e1325b"}},{"code_sha256_prefix":"b6d6bcb982b768cc","entry":"get_logger","repo":"cambridgeltl/autopeft","repo_kind":"official","path":"logger.py","file_url":"https://github.com/cambridgeltl/autopeft/blob/HEAD/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"b6d6bcb982b768cc"}},{"code_sha256_prefix":"cc8c8c3ebf0c343f","entry":"mish","repo":"cambridgeltl/autopeft","repo_kind":"official","path":"adapter-transformers-adapters3.1.0/src/transformers/activations_tf.py","file_url":"https://github.com/cambridgeltl/autopeft/blob/HEAD/adapter-transformers-adapters3.1.0/src/transformers/activations_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"cc8c8c3ebf0c343f"}},{"code_sha256_prefix":"e2d56cb91f999bef","entry":"quick_gelu","repo":"cambridgeltl/autopeft","repo_kind":"official","path":"adapter-transformers-adapters3.1.0/src/transformers/activations_tf.py","file_url":"https://github.com/cambridgeltl/autopeft/blob/HEAD/adapter-transformers-adapters3.1.0/src/transformers/activations_tf.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"e2d56cb91f999bef"}},{"code_sha256_prefix":"83eba990d34db6a2","entry":"simple_accuracy","repo":"cambridgeltl/autopeft","repo_kind":"official","path":"glue_metrics.py","file_url":"https://github.com/cambridgeltl/autopeft/blob/HEAD/glue_metrics.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"83eba990d34db6a2"}},{"code_sha256_prefix":"f247d1c0140d8564","entry":"split_datasets","repo":"cambridgeltl/autopeft","repo_kind":"official","path":"adapterhub/nas_search.py","file_url":"https://github.com/cambridgeltl/autopeft/blob/HEAD/adapterhub/nas_search.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"f247d1c0140d8564"}},{"code_sha256_prefix":"c2b46bdd4073aaa8","entry":"split_datasets","repo":"cambridgeltl/autopeft","repo_kind":"official","path":"adapterhub/run_glue.py","file_url":"https://github.com/cambridgeltl/autopeft/blob/HEAD/adapterhub/run_glue.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"c2b46bdd4073aaa8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}