Methods › General › Attention Modules › Multi-Head Attention
Multi-Head Attention
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Multi-head Attention is a module for attention mechanisms which runs through an attention mechanism several times in parallel. The independent attention outputs are then concatenated and linearly transformed into the expected dimension. Intuitively, multiple attention heads allows for attending to parts of the sequence differently (e.g. longer-term dependencies versus shorter-term dependencies).
MultiHead(Q, K, V) = [head₁,…,headₕ]W₀
where headᵢ = Attention (QWᵢ^Q, KWᵢᴷ, VWᵢⱽ )
Above W are all learnable parameter matrices.
Note that scaled dot-product attention is most commonly used in this module, although in principle it can be swapped out for other types of attention mechanism.
Source: Lilian Weng
Papers archive 2025-07-28
30 shown of 24,855, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
Augmenting Large Language Models with Static Code Analysis for Automated Code Quality Improvements 12 Jun 2025 · 0 repositories · arXiv:2506.10330
-
CIIR@LiveRAG 2025: Optimizing Multi-Agent Retrieval Augmented Generation through Self-Training 12 Jun 2025 · 1 repository · arXiv:2506.10844
-
Constructing and Evaluating Declarative RAG Pipelines in PyTerrier 12 Jun 2025 · 1 repository · arXiv:2506.10802
-
DART: Differentiable Dynamic Adaptive Region Tokenizer for Vision Transformer and Mamba 12 Jun 2025 · 1 repository · arXiv:2506.10390
-
Decomposing MLP Activations into Interpretable Features via Semi-Nonnegative Matrix Factorization 12 Jun 2025 · 1 repository · arXiv:2506.10920
-
NeuralNexus at BEA 2025 Shared Task: Retrieval-Augmented Prompting for Mistake Identification in AI Tutors 12 Jun 2025 · 1 repository · arXiv:2506.10627
-
PiPViT: Patch-based Visual Interpretable Prototypes for Retinal Image Analysis 12 Jun 2025 · 1 repository · arXiv:2506.10669
-
Reasoning RAG via System 1 or System 2: A Survey on Reasoning Agentic Retrieval-Augmented Generation for Industry Challenges 12 Jun 2025 · 0 repositories · arXiv:2506.10408
-
TableRAG: A Retrieval Augmented Generation Framework for Heterogeneous Document Reasoning 12 Jun 2025 · 1 repository · arXiv:2506.10380
-
Towards Robust Multimodal Emotion Recognition under Missing Modalities and Distribution Shifts 12 Jun 2025 · 1 repository · arXiv:2506.10452
-
Retrieval of Surface Solar Radiation through Implicit Albedo Recovery from Temporal Context 11 Jun 2025 · 1 repository · arXiv:2506.10174
-
A Novel Lightweight Transformer with Edge-Aware Fusion for Remote Sensing Image Captioning 11 Jun 2025 · 0 repositories · arXiv:2506.09429
-
Auto-Compressing Networks 11 Jun 2025 · 0 repositories · arXiv:2506.09714
-
Learning Efficient and Generalizable Graph Retriever for Knowledge-Graph Question Answering 11 Jun 2025 · 1 repository · arXiv:2506.09645
-
LLM-Powered CPI Prediction Inference with Online Text Time Series 11 Jun 2025 · 1 repository · arXiv:2506.09516
-
Measuring Corporate Human Capital Disclosures: Lexicon, Data, Code, and Research Opportunities 11 Jun 2025 · 0 repositories · arXiv:2506.10155
-
SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot 11 Jun 2025 · 0 repositories · arXiv:2506.09613
-
You Are What You Say: Exploiting Linguistic Content for VoicePrivacy Attacks 11 Jun 2025 · 0 repositories · arXiv:2506.09521
-
Safeguarding Multimodal Knowledge Copyright in the RAG-as-a-Service Environment 10 Jun 2025 · 1 repository · arXiv:2506.10030
-
AraReasoner: Evaluating Reasoning-Based LLMs for Arabic NLP 10 Jun 2025 · 0 repositories · arXiv:2506.08768
-
BioLangFusion: Multimodal Fusion of DNA, mRNA, and Protein Language Models 10 Jun 2025 · 0 repositories · arXiv:2506.08936
-
CC-RAG: Structured Multi-Hop Reasoning via Theme-Based Causal Graphs 10 Jun 2025 · 0 repositories · arXiv:2506.08364
-
DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in Search-Augmented LLMs 10 Jun 2025 · 1 repository · arXiv:2506.08500
-
Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving 10 Jun 2025 · 1 repository · arXiv:2506.08349Syntology ran 3 of 3 samples · 0 unverified · 3 pointer-only (licence)
-
FaithfulRAG: Fact-Level Conflict Modeling for Context-Faithful Retrieval-Augmented Generation 10 Jun 2025 · 1 repository · arXiv:2506.08938
-
FedRAG: A Framework for Fine-Tuning Retrieval-Augmented Generation Systems 10 Jun 2025 · 1 repository · arXiv:2506.09200
-
Fine-Grained Spatially Varying Material Selection in Images 10 Jun 2025 · 0 repositories · arXiv:2506.09023
-
FloorplanMAE:A self-supervised framework for complete floorplan generation from partial inputs 10 Jun 2025 · 0 repositories · arXiv:2506.08363
-
Hierarchical Neural Collapse Detection Transformer for Class Incremental Object Detection 10 Jun 2025 · 0 repositories · arXiv:2506.08562
-
Hyperspectral Image Classification via Transformer-based Spectral-Spatial Attention Decoupling and Adaptive Gating 10 Jun 2025 · 1 repository · arXiv:2506.08324
Tasks archive 2025-07-28
20 shown of 2,604 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
| Task | Papers |
|---|---|
| Language Modelling | 3,000 |
| Language Modeling | 2,346 |
| Retrieval | 1,792 |
| Question Answering | 1,481 |
| Decoder | 1,392 |
| Sentence | 1,337 |
| RAG | 1,320 |
| Retrieval-augmented Generation | 1,149 |
| Translation | 1,035 |
| Machine Translation | 929 |
| Semantic Segmentation | 859 |
| Large Language Model | 836 |
| Text Generation | 762 |
| Transfer Learning | 713 |
| Image Classification | 709 |
| Object Detection | 665 |
| Representation Learning | 630 |
| Sentiment Analysis | 614 |
| Classification | 607 |
| Segmentation | 596 |
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections