Papers › X-Linear Attention Networks for Image Captioning

X-Linear Attention Networks for Image Captioning

31 Mar 2020CVPR 2020 6arXiv:2003.14080archive 2025-07-28

Yingwei Pan, Ting Yao, Yehao Li, Tao Mei

Recent progress on fine-grained visual recognition and visual question answering has featured Bilinear Pooling, which effectively models the 2ⁿᵈ order interactions across multi-modal inputs. Nevertheless, there has not been evidence in support of building such interactions concurrently with attention mechanism for image captioning. In this paper, we introduce a unified attention block -- X-Linear attention block, that fully employs bilinear pooling to selectively capitalize on visual information or perform multi-modal reasoning. Technically, X-Linear attention block simultaneously exploits both the spatial and channel-wise bilinear attention distributions to capture the 2ⁿᵈ order interactions between the input single-modal or multi-modal features. Higher and even infinity order feature interactions are readily modeled through stacking multiple X-Linear attention blocks and equipping the block with Exponential Linear Unit (ELU) in a parameter-free fashion, respectively. Furthermore, we present X-Linear Attention Networks (dubbed as X-LAN) that novelly integrates X-Linear attention block(s) into image encoder and sentence decoder of image captioning model to leverage higher order intra- and inter-modal interactions. The experiments on COCO benchmark demonstrate that our X-LAN obtains to-date the best published CIDEr performance of 132.0% on COCO Karpathy test split. When further endowing Transformer with X-Linear attention blocks, CIDEr is boosted up to 132.8%. Source code is available at \url{https://github.com/Panda-Peter/image-captioning}.

Paper PDF Conference PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

Panda-Peter/image-captioning officialmentioned in papermentioned on GitHubpytorch report

jdai-cv/image-captioning mentioned on GitHubpytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Decoder Fine-Grained Visual Recognition Image Captioning Question Answering Sentence Visual Question Answering Visual Question Answering (VQA)

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Image Captioning	COCO Captions	X-Transformer	BLEU-1	80.9	#22 of 41	Archive leaderboard	report
Image Captioning	COCO Captions	X-Transformer	BLEU-2	65.8	#22 of 41	Archive leaderboard	report
Image Captioning	COCO Captions	X-Transformer	BLEU-3	51.5	#22 of 41	Archive leaderboard	report
Image Captioning	COCO Captions	X-Transformer	BLEU-4	39.7	#22 of 41	Archive leaderboard	report
Image Captioning	COCO Captions	X-Transformer	CIDER	132.8	#22 of 41	Archive leaderboard	report
Image Captioning	COCO Captions	X-Transformer	METEOR	29.5	#22 of 41	Archive leaderboard	report
Image Captioning	COCO Captions	X-Transformer	ROUGE-L	59.1	#22 of 41	Archive leaderboard	report
Image Captioning	COCO Captions	X-Transformer	SPICE	23.4	#22 of 41	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Absolute Position Encodings Adam Attention BPE Dense Connections Dropout Label Smoothing Layer Normalization Linear Layer Multi-Head Attention Position-Wise Feed-Forward Layer ReLU Residual Connection Softmax Transformer

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections