Methods › General › Attention Modules › Multi-Head Attention
Multi-Head Attention
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Multi-head Attention is a module for attention mechanisms which runs through an attention mechanism several times in parallel. The independent attention outputs are then concatenated and linearly transformed into the expected dimension. Intuitively, multiple attention heads allows for attending to parts of the sequence differently (e.g. longer-term dependencies versus shorter-term dependencies).
MultiHead(Q, K, V) = [head₁,…,headₕ]W₀
where headᵢ = Attention (QWᵢ^Q, KWᵢᴷ, VWᵢⱽ )
Above W are all learnable parameter matrices.
Note that scaled dot-product attention is most commonly used in this module, although in principle it can be swapped out for other types of attention mechanism.
Source: Lilian Weng
In Syntology name not resolved (unknown_name). Other entries in Syntology's graph carry this name; the lookup's note below says what that means for this name.
From the lookup's note on this name: Nothing resolved to this name, but 2 entries in Syntology's graph carry it; the most cited is the paper “Multi-Head Attention: Collaborate Instead of Concatenate” (arXiv:2006.16362, 191 citations). The lookup reads this as a miss of its own name matching, not a gap in coverage: the graph holds the name under a spelling the lookup did not reach. For arXiv:2006.16362: its page on this site · ask the lookup about it.
Near matches the lookup lists, each a different name: Multi-Head Knowledge Attention; multi-head attention neural network; Multi-head Explicit Attention (MEA); Gaussian Multi-head Attention (GMA); Multihead Low-Rank Attention (MLRA); Multi-head Attention Fusion Network; Multi-Head Low-Rank Attention (MLRA); Multi-Head Tropical Attention (MHTA).
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 24,855 · with a code link: 11,214 · where Syntology ran a sample: 3,454 (2,813 with a run with no instrument failure, 641 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (3,454 of 24,855 tagged: 2,813 with a run with no instrument failure, 641 where every run was a failure of Syntology's instrument)
Page 1 of 249: papers 1 to 100 of 24,855, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
Augmenting Large Language Models with Static Code Analysis for Automated Code Quality Improvements 12 Jun 2025 · 0 repositories · arXiv:2506.10330
-
CIIR@LiveRAG 2025: Optimizing Multi-Agent Retrieval Augmented Generation through Self-Training 12 Jun 2025 · 1 repository · arXiv:2506.10844
-
Constructing and Evaluating Declarative RAG Pipelines in PyTerrier 12 Jun 2025 · 1 repository · arXiv:2506.10802
-
DART: Differentiable Dynamic Adaptive Region Tokenizer for Vision Transformer and Mamba 12 Jun 2025 · 1 repository · arXiv:2506.10390
-
Decomposing MLP Activations into Interpretable Features via Semi-Nonnegative Matrix Factorization 12 Jun 2025 · 1 repository · arXiv:2506.10920
-
NeuralNexus at BEA 2025 Shared Task: Retrieval-Augmented Prompting for Mistake Identification in AI Tutors 12 Jun 2025 · 1 repository · arXiv:2506.10627
-
PiPViT: Patch-based Visual Interpretable Prototypes for Retinal Image Analysis 12 Jun 2025 · 1 repository · arXiv:2506.10669
-
Reasoning RAG via System 1 or System 2: A Survey on Reasoning Agentic Retrieval-Augmented Generation for Industry Challenges 12 Jun 2025 · 0 repositories · arXiv:2506.10408
-
TableRAG: A Retrieval Augmented Generation Framework for Heterogeneous Document Reasoning 12 Jun 2025 · 1 repository · arXiv:2506.10380
-
Towards Robust Multimodal Emotion Recognition under Missing Modalities and Distribution Shifts 12 Jun 2025 · 1 repository · arXiv:2506.10452
-
Retrieval of Surface Solar Radiation through Implicit Albedo Recovery from Temporal Context 11 Jun 2025 · 1 repository · arXiv:2506.10174
-
A Novel Lightweight Transformer with Edge-Aware Fusion for Remote Sensing Image Captioning 11 Jun 2025 · 0 repositories · arXiv:2506.09429
-
Auto-Compressing Networks 11 Jun 2025 · 0 repositories · arXiv:2506.09714
-
Learning Efficient and Generalizable Graph Retriever for Knowledge-Graph Question Answering 11 Jun 2025 · 1 repository · arXiv:2506.09645
-
LLM-Powered CPI Prediction Inference with Online Text Time Series 11 Jun 2025 · 1 repository · arXiv:2506.09516
-
Measuring Corporate Human Capital Disclosures: Lexicon, Data, Code, and Research Opportunities 11 Jun 2025 · 0 repositories · arXiv:2506.10155
-
SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot 11 Jun 2025 · 0 repositories · arXiv:2506.09613
-
You Are What You Say: Exploiting Linguistic Content for VoicePrivacy Attacks 11 Jun 2025 · 0 repositories · arXiv:2506.09521
-
Safeguarding Multimodal Knowledge Copyright in the RAG-as-a-Service Environment 10 Jun 2025 · 1 repository · arXiv:2506.10030
-
AraReasoner: Evaluating Reasoning-Based LLMs for Arabic NLP 10 Jun 2025 · 0 repositories · arXiv:2506.08768
-
BioLangFusion: Multimodal Fusion of DNA, mRNA, and Protein Language Models 10 Jun 2025 · 0 repositories · arXiv:2506.08936
-
CC-RAG: Structured Multi-Hop Reasoning via Theme-Based Causal Graphs 10 Jun 2025 · 0 repositories · arXiv:2506.08364
-
DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in Search-Augmented LLMs 10 Jun 2025 · 1 repository · arXiv:2506.08500
-
Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving 10 Jun 2025 · 1 repository · arXiv:2506.08349Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
FaithfulRAG: Fact-Level Conflict Modeling for Context-Faithful Retrieval-Augmented Generation 10 Jun 2025 · 1 repository · arXiv:2506.08938
-
FedRAG: A Framework for Fine-Tuning Retrieval-Augmented Generation Systems 10 Jun 2025 · 1 repository · arXiv:2506.09200
-
Fine-Grained Spatially Varying Material Selection in Images 10 Jun 2025 · 0 repositories · arXiv:2506.09023
-
FloorplanMAE:A self-supervised framework for complete floorplan generation from partial inputs 10 Jun 2025 · 0 repositories · arXiv:2506.08363
-
Hierarchical Neural Collapse Detection Transformer for Class Incremental Object Detection 10 Jun 2025 · 0 repositories · arXiv:2506.08562
-
Hyperspectral Image Classification via Transformer-based Spectral-Spatial Attention Decoupling and Adaptive Gating 10 Jun 2025 · 1 repository · arXiv:2506.08324
-
MedMoE: Modality-Specialized Mixture of Experts for Medical Vision-Language Understanding 10 Jun 2025 · 0 repositories · arXiv:2506.08356
-
Olica: Efficient Structured Pruning of Large Language Models without Retraining 10 Jun 2025 · 1 repository · arXiv:2506.08436Syntology official (archive's flag): 4 ran · 4 ran (of which 2 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 6 unverified (of 10 harvested samples) · 10 pointer-only (licence)
-
PatchGuard: Adversarially Robust Anomaly Detection and Localization through Vision Transformers and Pseudo Anomalies 10 Jun 2025 · 2 repositories · arXiv:2506.09237
-
Robust Visual Localization via Semantic-Guided Multi-Scale Transformer 10 Jun 2025 · 0 repositories · arXiv:2506.08526
-
TACTIC: Translation Agents with Cognitive-Theoretic Interactive Collaboration 10 Jun 2025 · 1 repository · arXiv:2506.08403
-
The Predictive Brain: Neural Correlates of Word Expectancy Align with Large Language Model Prediction Probabilities 10 Jun 2025 · 0 repositories · arXiv:2506.08511
-
Time Series Representations for Classification Lie Hidden in Pretrained Vision Transformers 10 Jun 2025 · 0 repositories · arXiv:2506.08641
-
4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos 9 Jun 2025 · 0 repositories · arXiv:2506.08015
-
A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation 9 Jun 2025 · 0 repositories · arXiv:2506.08210
-
Generative Voice Bursts during Phone Call 9 Jun 2025 · 0 repositories · arXiv:2506.07526
-
Hierarchical Lexical Graph for Enhanced Multi-Hop Retrieval 9 Jun 2025 · 1 repository · arXiv:2506.08074
-
Lightweight Sequential Transformers for Blood Glucose Level Prediction in Type-1 Diabetes 9 Jun 2025 · 0 repositories · arXiv:2506.07864
-
LlamaRec-LKG-RAG: A Single-Pass, Learnable Knowledge Graph-RAG Framework for LLM-Based Ranking 9 Jun 2025 · 1 repository · arXiv:2506.07449
-
LLM-driven Indoor Scene Layout Generation via Scaled Human-aligned Data Synthesis and Multi-Stage Preference Optimization 9 Jun 2025 · 0 repositories · arXiv:2506.07570
-
MADFormer: Mixed Autoregressive and Diffusion Transformers for Continuous Image Generation 9 Jun 2025 · 0 repositories · arXiv:2506.07999
-
Multilingual Hate Speech Detection in Social Media Using Translation-Based Approaches with Large Language Models 9 Jun 2025 · 0 repositories · arXiv:2506.08147
-
Quantum Graph Transformer for NLP Sentiment Classification 9 Jun 2025 · 0 repositories · arXiv:2506.07937
-
SceneRAG: Scene-level Retrieval-Augmented Generation for Video Understanding 9 Jun 2025 · 0 repositories · arXiv:2506.07600
-
Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Models 8 Jun 2025 · 0 repositories · arXiv:2506.07121
-
Breaking Data Silos: Towards Open and Scalable Mobility Foundation Models via Generative Continual Learning 7 Jun 2025 · 0 repositories · arXiv:2506.06694
-
Can In-Context Reinforcement Learning Recover From Reward Poisoning Attacks? 7 Jun 2025 · 0 repositories · arXiv:2506.06891
-
RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation 7 Jun 2025 · 0 repositories · arXiv:2506.06677
-
BEAST: Efficient Tokenization of B-Splines Encoded Action Sequences for Imitation Learning 6 Jun 2025 · 0 repositories · arXiv:2506.06072
-
Direct Behavior Optimization: Unlocking the Potential of Lightweight LLMs 6 Jun 2025 · 0 repositories · arXiv:2506.06401
-
Joint-GCG: Unified Gradient-Based Poisoning Attacks on Retrieval-Augmented Generation Systems 6 Jun 2025 · 1 repository · arXiv:2506.06151
-
Textile Analysis for Recycling Automation using Transfer Learning and Zero-Shot Foundation Models 6 Jun 2025 · 0 repositories · arXiv:2506.06569
-
The Lock-in Hypothesis: Stagnation by Algorithm 6 Jun 2025 · 0 repositories · arXiv:2506.06166
-
When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation 6 Jun 2025 · 1 repository · arXiv:2506.05690Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 5 harvested samples)
-
DACN: Dual-Attention Convolutional Network for Hyperspectral Image Super-Resolution 5 Jun 2025 · 1 repository · arXiv:2506.05041
-
A Multi-Dataset Evaluation of Models for Automated Vulnerability Repair 5 Jun 2025 · 0 repositories · arXiv:2506.04987
-
Design of intelligent proofreading system for English translation based on CNN and BERT 5 Jun 2025 · 0 repositories · arXiv:2506.04811
-
Dissecting Bias in LLMs: A Mechanistic Interpretability Perspective 5 Jun 2025 · 0 repositories · arXiv:2506.05166
-
Dynamic Context Tuning for Retrieval-Augmented Generation: Enhancing Multi-Turn Planning and Tool Adaptation 5 Jun 2025 · 0 repositories · arXiv:2506.11092
-
ECoRAG: Evidentiality-guided Compression for Long Context RAG 5 Jun 2025 · 1 repository · arXiv:2506.05167
-
Grapheme-Coherent Phonemic and Prosodic Annotation of Speech by Implicit and Explicit Grapheme Conditioning 5 Jun 2025 · 0 repositories · arXiv:2506.04527
-
Intentionally Unintentional: GenAI Exceptionalism and the First Amendment 5 Jun 2025 · 0 repositories · arXiv:2506.05211
-
Knowledgeable-r1: Policy Optimization for Knowledge Exploration in Retrieval-Augmented Generation 5 Jun 2025 · 1 repository · arXiv:2506.05154Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified (of 5 harvested samples)
-
Mathematical Reasoning for Unmanned Aerial Vehicles: A RAG-Based Approach for Complex Arithmetic Reasoning 5 Jun 2025 · 1 repository · arXiv:2506.04998
-
Micro-Act: Mitigate Knowledge Conflict in Question Answering via Actionable Self-Reasoning 5 Jun 2025 · 1 repository · arXiv:2506.05278
-
On Automating Security Policies with Contemporary LLMs 5 Jun 2025 · 0 repositories · arXiv:2506.04838
-
Predicting ICU In-Hospital Mortality Using Adaptive Transformer Layer Fusion 5 Jun 2025 · 1 repository · arXiv:2506.04924
-
The NTNU System at the S&I Challenge 2025 SLA Open Track 5 Jun 2025 · 0 repositories · arXiv:2506.05121
-
Facts are Harder Than Opinions -- A Multilingual, Comparative Analysis of LLM-Based Fact-Checking Reliability 4 Jun 2025 · 0 repositories · arXiv:2506.03655
-
Magic Mushroom: A Customizable Benchmark for Fine-grained Analysis of Retrieval Noise Erosion in RAG Systems 4 Jun 2025 · 0 repositories · arXiv:2506.03901
-
Privacy and Security Threat for OpenAI GPTs 4 Jun 2025 · 0 repositories · arXiv:2506.04036
-
R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning 4 Jun 2025 · 1 repository · arXiv:2506.04185
-
TracLLM: A Generic Framework for Attributing Long Context LLMs 4 Jun 2025 · 1 repository · arXiv:2506.04202
-
Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Bounds 3 Jun 2025 · 0 repositories · arXiv:2506.03100
-
An Exploratory Framework for Future SETI Applications: Detecting Generative Reactivity via Language Models 3 Jun 2025 · 0 repositories · arXiv:2506.02730
-
Attacking Attention of Foundation Models Disrupts Downstream Tasks 3 Jun 2025 · 1 repository · arXiv:2506.05394
-
CoRe-MMRAG: Cross-Source Knowledge Reconciliation for Multimodal RAG 3 Jun 2025 · 0 repositories · arXiv:2506.02544
-
Enhancing Automatic PT Tagging for MEDLINE Citations Using Transformer-Based Models 3 Jun 2025 · 0 repositories · arXiv:2506.03321
-
Multi-modal brain MRI synthesis based on SwinUNETR 3 Jun 2025 · 0 repositories · arXiv:2506.02467
-
Rethinking the effects of data contamination in Code Intelligence 3 Jun 2025 · 0 repositories · arXiv:2506.02791
-
Talk2SAM: Text-Guided Semantic Enhancement for Complex-Shaped Object Segmentation 3 Jun 2025 · 0 repositories · arXiv:2506.05396
-
WeightLoRA: Keep Only Necessary Adapters 3 Jun 2025 · 0 repositories · arXiv:2506.02724
-
Automatic Stage Lighting Control: Is it a Rule-Driven Process or Generative Task? 2 Jun 2025 · 1 repository · arXiv:2506.01482
-
LLM in the Loop: Creating the PARADEHATE Dataset for Hate Speech Detoxification 2 Jun 2025 · 0 repositories · arXiv:2506.01484
-
LLMs as World Models: Data-Driven and Human-Centered Pre-Event Simulation for Disaster Impact Assessment 2 Jun 2025 · 0 repositories · arXiv:2506.06355
-
Retrieval-Augmented Generation of Ontologies from Relational Databases 2 Jun 2025 · 0 repositories · arXiv:2506.01232
-
A Graph-Retrieval-Augmented Generation Framework Enhances Decision-Making in the Circular Economy 1 Jun 2025 · 0 repositories · arXiv:2506.04252
-
CLAP-ART: Automated Audio Captioning with Semantic-rich Audio Representation Tokenizer 1 Jun 2025 · 0 repositories · arXiv:2506.00800
-
How Neural Networks Organize Concepts: Introducing Concept Trajectory Analysis for Deep Learning Interpretability 1 Jun 2025 · 1 repository
-
Leveraging AM and FM Rhythm Spectrograms for Dementia Classification and Assessment 1 Jun 2025 · 1 repository · arXiv:2506.00861
-
RARE: Retrieval-Aware Robustness Evaluation for Retrieval-Augmented Generation Systems 1 Jun 2025 · 1 repository · arXiv:2506.00789
-
Rhythm Controllable and Efficient Zero-Shot Voice Conversion via Shortcut Flow Matching 1 Jun 2025 · 0 repositories · arXiv:2506.01014
-
Assortment of Attention Heads: Accelerating Federated PEFT with Head Pruning and Strategic Client Selection 31 May 2025 · 0 repositories · arXiv:2506.00743
-
Channel-Imposed Fusion: A Simple yet Effective Method for Medical Time Series Classification 31 May 2025 · 0 repositories · arXiv:2506.00337
-
Evaluating Robot Policies in a World Model 31 May 2025 · 0 repositories · arXiv:2506.00613
-
FinBERT2: A Specialized Bidirectional Encoder for Bridging the Gap in Finance-Specific Deployment of Large Language Models 31 May 2025 · 0 repositories · arXiv:2506.06335
Tasks archive 2025-07-28
20 shown of 2,604 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
| Task | Papers |
|---|---|
| Language Modelling | 3,000 |
| Language Modeling | 2,346 |
| Retrieval | 1,792 |
| Question Answering | 1,481 |
| Decoder | 1,392 |
| Sentence | 1,337 |
| RAG | 1,320 |
| Retrieval-augmented Generation | 1,149 |
| Translation | 1,035 |
| Machine Translation | 929 |
| Semantic Segmentation | 859 |
| Large Language Model | 836 |
| Text Generation | 762 |
| Transfer Learning | 713 |
| Image Classification | 709 |
| Object Detection | 665 |
| Representation Learning | 630 |
| Sentiment Analysis | 614 |
| Classification | 607 |
| Segmentation | 596 |
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections