Methods › General › Attention Modules › Multi-DConv-Head Attention
Multi-DConv-Head Attention
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Multi-DConv-Head Attention, or MDHA, is a type of Multi-Head Attention that utilizes depthwise convolutions after the multi-head projections. It is used in the Primer Transformer architecture.
Specifically, 3x1 depthwise convolutions are added after each of the multi-head projections for query Q, key K and value V in self-attention. These depthwise convolutions are performed over the spatial dimension of each dense projection’s output. Interestingly, this ordering of pointwise followed by depthwise convolution is the reverse of typical separable convolution, which the authors find to be less effective. They also find that wider depthwise convolution and standard convolution not only do not improve performance, but in several cases hurt it.
MDHA is similar to Convolutional Attention, which uses separable convolution instead of depthwise convolution and does not apply convolution operations per attention head as in MDHA.
Papers archive 2025-07-28
15 shown of 15, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignment 20 May 2025 · 0 repositories · arXiv:2505.14667
-
A review of DNA restriction-free overlapping sequence cloning techniques for synthetic biology 6 May 2025 · 0 repositories · arXiv:2505.03681
-
Primer C-VAE: An interpretable deep learning primer design method to detect emerging virus variants 3 Mar 2025 · 0 repositories · arXiv:2503.01459
-
Characteristic Performance Study on Solving Oscillator ODEs via Soft-constrained Physics-informed Neural Network with Small Data 19 Aug 2024 · 1 repository · arXiv:2408.11077
-
The curious case of A31P, a topology-switching mutant of the Repressor of Primer protein : A molecular dynamics study of its folding and misfolding 1 Apr 2024 · 0 repositories · arXiv:2404.01405
-
RCMHA: Relative Convolutional Multi-Head Attention for Natural Language Modelling 7 Aug 2023 · 1 repository · arXiv:2308.03429
-
Brainformers: Trading Simplicity for Efficiency 29 May 2023 · 0 repositories · arXiv:2306.00008
-
The Effects of Political Martyrdom on Election Results: The Assassination of Abe 29 May 2023 · 1 repository · arXiv:2305.18004
-
Towards NeuroAI: Introducing Neuronal Diversity into Artificial Neural Networks 23 Jan 2023 · 0 repositories · arXiv:2301.09245
-
N-Grammer: Augmenting Transformers with latent n-grams 13 Jul 2022 · 2 repositories · arXiv:2207.06366Syntology ran 0 of 6 samples · 6 unverified
-
Piecewise Linear Neural Networks and Deep Learning 18 Jun 2022 · 0 repositories · arXiv:2206.09149
-
Enriching and Characterizing T-Cell Repertoires from 3' Barcoded Single-Cell Whole Transcriptome Amplification Products 21 Mar 2022 · 0 repositories · arXiv:2203.11266
-
Searching for Efficient Transformers for Language Modeling 1 Dec 2021 · 0 repositories
-
N-grammer: Augmenting Transformers with latent n-grams 16 Nov 2021 · 0 repositories
-
Primer: Searching for Efficient Transformers for Language Modeling 17 Sep 2021 · 4 repositories · arXiv:2109.08668Syntology ran 3 of 3 samples · 0 unverified · 3 pointer-only (licence)
Tasks archive 2025-07-28
18 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections