{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/llama-adapter-efficient-fine-tuning-of","title":"LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention","arxiv_id":"2303.16199","date":"2023-03-28","proceeding":null,"authors":["Renrui Zhang","Jiaming Han","Chris Liu","Peng Gao","Aojun Zhou","Xiangfei Hu","Shilin Yan","Pan Lu","Hongsheng Li","Yu Qiao"],"abstract":"We present LLaMA-Adapter, a lightweight adaption method to efficiently fine-tune LLaMA into an instruction-following model. Using 52K self-instruct demonstrations, LLaMA-Adapter only introduces 1.2M learnable parameters upon the frozen LLaMA 7B model, and costs less than one hour for fine-tuning on 8 A100 GPUs. Specifically, we adopt a set of learnable adaption prompts, and prepend them to the word tokens at higher transformer layers. Then, a zero-initialized attention mechanism with zero gating is proposed, which adaptively injects the new instructional cues into LLaMA, while effectively preserves its pre-trained knowledge. With our efficient training, LLaMA-Adapter can generate high-quality responses, comparable to Alpaca with fully fine-tuned 7B parameters. Besides language commands, our approach can be simply extended to multi-modal instructions for learning image-conditioned LLaMA model, which achieves superior reasoning performance on ScienceQA and COCO Caption benchmarks. Furthermore, we also evaluate the zero-initialized attention mechanism for fine-tuning other pre-trained models (ViT, RoBERTa) on traditional vision and language tasks, demonstrating the superior generalization capacity of our approach. Code is released at https://github.com/OpenGVLab/LLaMA-Adapter.","url_abs":"https://arxiv.org/abs/2303.16199v3","url_pdf":"https://arxiv.org/pdf/2303.16199v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"llama-adapter-efficient-fine-tuning-of","repo_url":"https://github.com/opengvlab/llama-adapter","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"GPL-3.0"}},{"paper_slug":"llama-adapter-efficient-fine-tuning-of","repo_url":"https://github.com/zrrskywalker/llama-adapter","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"llama-adapter-efficient-fine-tuning-of","repo_url":"https://github.com/Lightning-AI/lit-llama","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"llama-adapter-efficient-fine-tuning-of","repo_url":"https://github.com/alpha-vllm/llama2-accessory","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"llama-adapter-efficient-fine-tuning-of","repo_url":"https://github.com/zihanzhaosjtu/librisqa","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"llama-adapter-efficient-fine-tuning-of","repo_url":"https://github.com/ziyuguo99/point-bind_point-llm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"llama-adapter-efficient-fine-tuning-of","repo_url":"https://github.com/zrrskywalker/point-bind","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"multimodal-deep-learning","task_name":"Multimodal Deep Learning"},{"task_slug":"music-question-answering","task_name":"Music Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/music-question-answering-on-musicqa","task":"Music Question Answering","dataset":"MusicQA","model":"LLaMA Adapter","rank_in_archive_order":2,"of":3,"metrics":{"BERT Score":"0.895","BLEU":"0.273","METEOR":"0.334","ROUGE":"0.413"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2303.16199","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}