{"url":"/method/multi-head-attention","slug":"multi-head-attention","name":"Multi-Head Attention","full_name":"Multi-Head Attention","full_name_withheld":false,"description_markdown":"**Multi-head Attention** is a module for attention mechanisms which runs through an attention mechanism several times in parallel. The independent attention outputs are then concatenated and linearly transformed into the expected dimension. Intuitively, multiple attention heads allows for attending to parts of the sequence differently (e.g. longer-term dependencies versus shorter-term dependencies). \r\n\r\n$$ \\text{MultiHead}\\left(\\textbf{Q}, \\textbf{K}, \\textbf{V}\\right) = \\left[\\text{head}\\_{1},\\dots,\\text{head}\\_{h}\\right]\\textbf{W}_{0}$$\r\n\r\n$$\\text{where} \\text{ head}\\_{i} = \\text{Attention} \\left(\\textbf{Q}\\textbf{W}\\_{i}^{Q}, \\textbf{K}\\textbf{W}\\_{i}^{K}, \\textbf{V}\\textbf{W}\\_{i}^{V} \\right) $$\r\n\r\nAbove $\\textbf{W}$ are all learnable parameter matrices.\r\n\r\nNote that [scaled dot-product attention](https://paperswithcode.com/method/scaled) is most commonly used in this module, although in principle it can be swapped out for other types of attention mechanism.\r\n\r\nSource: [Lilian Weng](https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html#a-family-of-attention-mechanisms)","description_state":"present","introduced_year":null,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":"https://arxiv.org/abs/1706.03762v7","title":"Attention Is All You Need","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/fec78a687210851f055f792d45300d27cc60ae41/transformer/SubLayers.py#L9","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Attention Modules","url":"/methods/category/attention-modules","pwc_aliases":[]}],"n_papers_tagged":24855,"archive_num_papers":null,"papers_newest_first":[{"paper":null,"title":"Augmenting Large Language Models with Static Code Analysis for Automated Code Quality Improvements","date":"2025-06-12","arxiv_id":"2506.10330","n_code_links":0,"syntology":null},{"paper":"/paper/ciir-liverag-2025-optimizing-multi-agent","title":"CIIR@LiveRAG 2025: Optimizing Multi-Agent Retrieval Augmented Generation through Self-Training","date":"2025-06-12","arxiv_id":"2506.10844","n_code_links":1,"syntology":null},{"paper":"/paper/constructing-and-evaluating-declarative-rag","title":"Constructing and Evaluating Declarative RAG Pipelines in PyTerrier","date":"2025-06-12","arxiv_id":"2506.10802","n_code_links":1,"syntology":null},{"paper":"/paper/dart-differentiable-dynamic-adaptive-region","title":"DART: Differentiable Dynamic Adaptive Region Tokenizer for Vision Transformer and Mamba","date":"2025-06-12","arxiv_id":"2506.10390","n_code_links":1,"syntology":null},{"paper":"/paper/decomposing-mlp-activations-into","title":"Decomposing MLP Activations into Interpretable Features via Semi-Nonnegative Matrix Factorization","date":"2025-06-12","arxiv_id":"2506.10920","n_code_links":1,"syntology":null},{"paper":"/paper/neuralnexus-at-bea-2025-shared-task-retrieval","title":"NeuralNexus at BEA 2025 Shared Task: Retrieval-Augmented Prompting for Mistake Identification in AI Tutors","date":"2025-06-12","arxiv_id":"2506.10627","n_code_links":1,"syntology":null},{"paper":"/paper/pipvit-patch-based-visual-interpretable","title":"PiPViT: Patch-based Visual Interpretable Prototypes for Retinal Image Analysis","date":"2025-06-12","arxiv_id":"2506.10669","n_code_links":1,"syntology":null},{"paper":null,"title":"Reasoning RAG via System 1 or System 2: A Survey on Reasoning Agentic Retrieval-Augmented Generation for Industry Challenges","date":"2025-06-12","arxiv_id":"2506.10408","n_code_links":0,"syntology":null},{"paper":"/paper/tablerag-a-retrieval-augmented-generation","title":"TableRAG: A Retrieval Augmented Generation Framework for Heterogeneous Document Reasoning","date":"2025-06-12","arxiv_id":"2506.10380","n_code_links":1,"syntology":null},{"paper":"/paper/towards-robust-multimodal-emotion-recognition","title":"Towards Robust Multimodal Emotion Recognition under Missing Modalities and Distribution Shifts","date":"2025-06-12","arxiv_id":"2506.10452","n_code_links":1,"syntology":null},{"paper":"/paper/2506-10174","title":"Retrieval of Surface Solar Radiation through Implicit Albedo Recovery from Temporal Context","date":"2025-06-11","arxiv_id":"2506.10174","n_code_links":1,"syntology":null},{"paper":null,"title":"A Novel Lightweight Transformer with Edge-Aware Fusion for Remote Sensing Image Captioning","date":"2025-06-11","arxiv_id":"2506.09429","n_code_links":0,"syntology":null},{"paper":null,"title":"Auto-Compressing Networks","date":"2025-06-11","arxiv_id":"2506.09714","n_code_links":0,"syntology":null},{"paper":"/paper/learning-efficient-and-generalizable-graph","title":"Learning Efficient and Generalizable Graph Retriever for Knowledge-Graph Question Answering","date":"2025-06-11","arxiv_id":"2506.09645","n_code_links":1,"syntology":null},{"paper":"/paper/llm-powered-cpi-prediction-inference-with","title":"LLM-Powered CPI Prediction Inference with Online Text Time Series","date":"2025-06-11","arxiv_id":"2506.09516","n_code_links":1,"syntology":null},{"paper":null,"title":"Measuring Corporate Human Capital Disclosures: Lexicon, Data, Code, and Research Opportunities","date":"2025-06-11","arxiv_id":"2506.10155","n_code_links":0,"syntology":null},{"paper":null,"title":"SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot","date":"2025-06-11","arxiv_id":"2506.09613","n_code_links":0,"syntology":null},{"paper":null,"title":"You Are What You Say: Exploiting Linguistic Content for VoicePrivacy Attacks","date":"2025-06-11","arxiv_id":"2506.09521","n_code_links":0,"syntology":null},{"paper":"/paper/2506-10030","title":"Safeguarding Multimodal Knowledge Copyright in the RAG-as-a-Service Environment","date":"2025-06-10","arxiv_id":"2506.10030","n_code_links":1,"syntology":null},{"paper":null,"title":"AraReasoner: Evaluating Reasoning-Based LLMs for Arabic NLP","date":"2025-06-10","arxiv_id":"2506.08768","n_code_links":0,"syntology":null},{"paper":null,"title":"BioLangFusion: Multimodal Fusion of DNA, mRNA, and Protein Language Models","date":"2025-06-10","arxiv_id":"2506.08936","n_code_links":0,"syntology":null},{"paper":null,"title":"CC-RAG: Structured Multi-Hop Reasoning via Theme-Based Causal Graphs","date":"2025-06-10","arxiv_id":"2506.08364","n_code_links":0,"syntology":null},{"paper":"/paper/dragged-into-conflicts-detecting-and","title":"DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in Search-Augmented LLMs","date":"2025-06-10","arxiv_id":"2506.08500","n_code_links":1,"syntology":null},{"paper":"/paper/evaluating-llms-across-multi-cognitive-levels","title":"Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving","date":"2025-06-10","arxiv_id":"2506.08349","n_code_links":1,"syntology":{"ran":3,"of":3,"unverified":0,"pointer_only":3}},{"paper":"/paper/faithfulrag-fact-level-conflict-modeling-for","title":"FaithfulRAG: Fact-Level Conflict Modeling for Context-Faithful Retrieval-Augmented Generation","date":"2025-06-10","arxiv_id":"2506.08938","n_code_links":1,"syntology":null},{"paper":"/paper/fedrag-a-framework-for-fine-tuning-retrieval","title":"FedRAG: A Framework for Fine-Tuning Retrieval-Augmented Generation Systems","date":"2025-06-10","arxiv_id":"2506.09200","n_code_links":1,"syntology":null},{"paper":null,"title":"Fine-Grained Spatially Varying Material Selection in Images","date":"2025-06-10","arxiv_id":"2506.09023","n_code_links":0,"syntology":null},{"paper":null,"title":"FloorplanMAE:A self-supervised framework for complete floorplan generation from partial inputs","date":"2025-06-10","arxiv_id":"2506.08363","n_code_links":0,"syntology":null},{"paper":null,"title":"Hierarchical Neural Collapse Detection Transformer for Class Incremental Object Detection","date":"2025-06-10","arxiv_id":"2506.08562","n_code_links":0,"syntology":null},{"paper":"/paper/hyperspectral-image-classification-via","title":"Hyperspectral Image Classification via Transformer-based Spectral-Spatial Attention Decoupling and Adaptive Gating","date":"2025-06-10","arxiv_id":"2506.08324","n_code_links":1,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/language-modelling","name":"Language Modelling","papers":3000},{"task":"/task/language-modeling","name":"Language Modeling","papers":2346},{"task":"/task/retrieval","name":"Retrieval","papers":1792},{"task":"/task/question-answering","name":"Question Answering","papers":1481},{"task":"/task/decoder","name":"Decoder","papers":1392},{"task":"/task/sentence","name":"Sentence","papers":1337},{"task":"/task/rag","name":"RAG","papers":1320},{"task":"/task/retrieval-augmented-generation","name":"Retrieval-augmented Generation","papers":1149},{"task":"/task/translation","name":"Translation","papers":1035},{"task":"/task/machine-translation","name":"Machine Translation","papers":929},{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":859},{"task":"/task/large-language-model","name":"Large Language Model","papers":836},{"task":"/task/text-generation","name":"Text Generation","papers":762},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":713},{"task":"/task/image-classification","name":"Image Classification","papers":709},{"task":"/task/object-detection","name":"Object Detection","papers":665},{"task":"/task/representation-learning","name":"Representation Learning","papers":630},{"task":"/task/sentiment-analysis","name":"Sentiment Analysis","papers":614},{"task":"/task/classification-1","name":"Classification","papers":607},{"task":"/task/segmentation","name":"Segmentation","papers":596}],"tasks_shown":20,"n_tasks":2604,"usage_by_year":[{"year":"2017","papers":22},{"year":"2018","papers":126},{"year":"2019","papers":1081},{"year":"2020","papers":2162},{"year":"2021","papers":3154},{"year":"2022","papers":3332},{"year":"2023","papers":5274},{"year":"2024","papers":7098},{"year":"2025","papers":2606}],"row_source":"embedded","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/multi-head-attention"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}