Papers › SILC: Improving Vision Language Pretraining with Self-Distillation

SILC: Improving Vision Language Pretraining with Self-Distillation

20 Oct 2023arXiv:2310.13355archive 2025-07-28

Muhammad Ferjad Naeem, Yongqin Xian, Xiaohua Zhai, Lukas Hoyer, Luc van Gool, Federico Tombari

Image-Text pretraining on web-scale image caption datasets has become the default recipe for open vocabulary classification and retrieval models thanks to the success of CLIP and its variants. Several works have also used CLIP features for dense prediction tasks and have shown the emergence of open-set abilities. However, the contrastive objective used by these models only focuses on image-text alignment and does not incentivise image feature learning for dense prediction tasks. In this work, we introduce SILC, a novel framework for vision language pretraining. SILC improves image-text contrastive learning with the simple addition of local-to-global correspondence learning by self-distillation. We show that distilling local image features from an exponential moving average (EMA) teacher model significantly improves model performance on dense predictions tasks like detection and segmentation, while also providing improvements on image-level tasks such as classification and retrieval. SILC models sets a new state of the art for zero-shot classification, few shot classification, image and text retrieval, zero-shot segmentation, and open vocabulary segmentation. We further show that SILC features greatly benefit open vocabulary detection, captioning and visual question answering.

Paper PDF

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

No code repository is listed for this paper in the archive or in Syntology's graph.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Classification Contrastive Learning Open Vocabulary Semantic Segmentation Question Answering Retrieval Segmentation Text Retrieval Visual Question Answering Zero Shot Segmentation Zero-Shot Learning

1 archive task tag without a task page not shown.

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Open Vocabulary Semantic Segmentation	ADE20K-150	SILC	mIoU	37.7	#5 of 23	Archive leaderboard	report
Open Vocabulary Semantic Segmentation	ADE20K-847	SILC	mIoU	15.0	#7 of 19	Archive leaderboard	report
Open Vocabulary Semantic Segmentation	PASCAL Context-459	SILC	mIoU	25.8	#1 of 15	Archive leaderboard	report
Open Vocabulary Semantic Segmentation	PASCAL Context-59	SILC	mIoU	63.5	#2 of 24	Archive leaderboard	report
Open Vocabulary Semantic Segmentation	PascalVOC-20	SILC	mIoU	97.6	#2 of 20	Archive leaderboard	report
Open Vocabulary Semantic Segmentation	PascalVOC-20b	SILC	mIoU	82.5	#3 of 4	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

CLIP Contrastive Learning

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections