Papers › Meta-Adapters: Parameter Efficient Few-shot Fine-tuning through Meta-Learning

Meta-Adapters: Parameter Efficient Few-shot Fine-tuning through Meta-Learning

19 Sep 2022Proceedings of the First International Conference on Automated Machine Learning 2022 9archive 2025-07-28

Trapit Bansal, Salaheddin Alzubi, Tong Wang, Jay-Yoon Lee, Andrew McCallum

Consistent improvements in the representational capacity of large pre-trained transformers has made it increasingly viable to serve these models as shared priors that can be fine-tuned on a large number of downstream tasks. However, fine-tuning the entire model for every task of interest makes a copy of all the model parameters, rendering such scenarios highly impractical. Recently introduced Adapter methods propose a promising alternative, one where only a small number of additional parameters are introduced per task specifically for fine-tuning. However, Adapters often require large amounts of task-specific data for good performance and don’t work well in data-scarce few-shot scenarios. In this paper, we approach parameter-efficient fine-tuning in few-shot settings from a meta-learning perspective. We introduce Meta-Adapters, which are small blocks of meta-learned adapter layers inserted in a pre-trained model that re-purpose a frozen pre-trained model into a parameter-efficient few-shot learner. Meta-Adapters perform competitively with state-of-the-art few-shot learning methods that require full fine-tuning, while only fine-tuning 0.6% of the parameters. We evaluate Meta-Adapters along with multiple transfer learning baselines on an evaluation suite of 17 classification tasks and find that they improve few-shot accuracy by a large margin over competitive parameter-efficient methods, while requiring significantly lesser parameters for fine-tuning. Moreover, when comparing few-shot prompting of GPT-3 against few-shot fine-tuning with Meta-Adapters, we find that Meta-Adapters perform competitively while working with pre-trained transformers that are many orders of magnitude (1590×) smaller in size than GPT-3.

PaperPDFCode

Syntology This paper has no arXiv id in the archive, and this site joins papers to Syntology's graph by arXiv id, so this page does not say whether the graph holds it.

Code

theTB/meta_adapters mentioned in papertfSyntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Few-Shot LearningMeta-LearningTransfer Learningparameter-efficient fine-tuning

Results from the paper archive 2025-07-28

No leaderboard rows for this paper in the archive.

Methods

AdamAdapterAttentionAttention DropoutBPECosine AnnealingDense ConnectionsDropoutGPT-3Layer NormalizationLinear LayerLinear Warmup With Cosine AnnealingMulti-Head AttentionResidual ConnectionSoftmaxWeight Decay

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections