Papers › SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentation

SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentation

30 Sep 2022CVPR 2023 1arXiv:2209.15323archive 2025-07-28

Rita Ramos, Bruno Martins, Desmond Elliott, Yova Kementchedjhieva

Recent advances in image captioning have focused on scaling the data and model size, substantially increasing the cost of pre-training and finetuning. As an alternative to large models, we present SmallCap, which generates a caption conditioned on an input image and related captions retrieved from a datastore. Our model is lightweight and fast to train, as the only learned parameters are in newly introduced cross-attention layers between a pre-trained CLIP encoder and GPT-2 decoder. SmallCap can transfer to new domains without additional finetuning and can exploit large-scale data in a training-free fashion since the contents of the datastore can be readily replaced. Our experiments show that SmallCap, trained only on COCO, has competitive performance on this benchmark, and also transfers to other domains without retraining, solely through retrieval from target-domain data. Further improvement is achieved through the training-free exploitation of diverse human-labeled and web data, which proves to be effective for a range of domains, including the nocaps benchmark, designed to test generalization to unseen visual concepts.

Paper PDF Conference PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

ritaramo/smallcap officialmentioned in paperpytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Decoder Image Captioning Retrieval

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Image Captioning	COCO Captions	SmallCapd=16, Large	BLEU-4	37.2	#29 of 41	Archive leaderboard	report
Image Captioning	COCO Captions	SmallCapd=16, Large	CIDER	121.8	#29 of 41	Archive leaderboard	report
Image Captioning	COCO Captions	SmallCapd=16, Large	METEOR	28.3	#29 of 41	Archive leaderboard	report
Image Captioning	COCO Captions	SmallCapd=16, Large	SPICE	21.5	#29 of 41	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Adam Attention Attention Dropout BPE CLIP Cosine Annealing Dense Connections Discriminative Fine-Tuning Dropout GPT-2 Layer Normalization Linear Layer Linear Warmup With Cosine Annealing Multi-Head Attention Residual Connection Softmax Test Weight Decay

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections