Papers › A Dot Product Attention Free Transformer

A Dot Product Attention Free Transformer

29 Sep 2021archive 2025-07-28

Shuangfei Zhai, Walter Talbott, Nitish Srivastava, Chen Huang, Hanlin Goh, Ruixiang Zhang, Joshua M. Susskind

We introduce Dot Product Attention Free Transformer (DAFT), an efficient variant of Transformers \citep{transformer} that eliminates the query-key dot product in self attention. The core idea is to construct a decomposable attention map for each dimension of the query, key and value. This compositionality enables an implementation where the attention tensor does not to be computed or stored explicitly. A DAFT layer has a memory complexity linear w.r.t. both the context size and the dimension of features, making it compatible with both large input and model sizes. We also introduce DAFT-conv, a model variant that takes advantage of locality and spatial weight sharing while maintaining global connectivity. We conduct experiments on ImageNet-1K classification, as well as CIFAR10 and Enwik8, two autoregressive modeling tasks. We show that DAFT demonstrates competitive performance on all the benchmarks, while providing excellent efficiency at the same time.

Paper PDF

Code

No code repository is listed for this paper in the archive or in Syntology's graph.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Image Classification Language Modelling

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Image Classification	ImageNet	DAFT-conv (384 heads, 300 epochs)	Number of params	23M	#682 of 1060	Archive leaderboard	report
Image Classification	ImageNet	DAFT-conv (384 heads, 300 epochs)	Top 1 Accuracy	80.8%	#682 of 1060	Archive leaderboard	report
Image Classification	ImageNet	DAFT-conv (16 heads)	Number of params	20.3M	#714 of 1060	Archive leaderboard	report
Image Classification	ImageNet	DAFT-conv (16 heads)	Top 1 Accuracy	80.2%	#714 of 1060	Archive leaderboard	report
Image Classification	ImageNet	DAFT-conv (384 heads, 200 epochs)	Number of params	23M	#718 of 1060	Archive leaderboard	report
Image Classification	ImageNet	DAFT-conv (384 heads, 200 epochs)	Top 1 Accuracy	80.1%	#718 of 1060	Archive leaderboard	report
Image Classification	ImageNet	DAFT-full	Number of params	22.6M	#742 of 1060	Archive leaderboard	report
Image Classification	ImageNet	DAFT-full	Top 1 Accuracy	79.8%	#742 of 1060	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Absolute Position Encodings Adam Attention Attention Free Transformer BPE Dense Connections Dropout Label Smoothing Layer Normalization Linear Layer Multi-Head Attention Position-Wise Feed-Forward Layer Residual Connection Softmax Transformer

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections