Papers › MDS-ViTNet: Improving saliency prediction for Eye-Tracking with Vision Transformer

MDS-ViTNet: Improving saliency prediction for Eye-Tracking with Vision Transformer

29 May 2024arXiv:2405.19501archive 2025-07-28

Polezhaev Ignat, Goncharenko Igor, Iurina Natalya

In this paper, we present a novel methodology we call MDS-ViTNet (Multi Decoder Saliency by Vision Transformer Network) for enhancing visual saliency prediction or eye-tracking. This approach holds significant potential for diverse fields, including marketing, medicine, robotics, and retail. We propose a network architecture that leverages the Vision Transformer, moving beyond the conventional ImageNet backbone. The framework adopts an encoder-decoder structure, with the encoder utilizing a Swin transformer to efficiently embed most important features. This process involves a Transfer Learning method, wherein layers from the Vision Transformer are converted by the Encoder Transformer and seamlessly integrated into a CNN Decoder. This methodology ensures minimal information loss from the original input image. The decoder employs a multi-decoding technique, utilizing dual decoders to generate two distinct attention maps. These maps are subsequently combined into a singular output via an additional CNN model. Our trained model MDS-ViTNet achieves state-of-the-art results across several benchmarks. Committed to fostering further collaboration, we intend to make our code, models, and datasets accessible to the public.

Paper PDF Code

Code

ignatpolezhaev/mds-vitnet officialmentioned in papermentioned on GitHubpytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Decoder Marketing Saliency Prediction Transfer Learning

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Saliency Prediction	SALICON	MDS-ViTNet	AUC	0.8684	#4 of 5	Archive leaderboard	report
Saliency Prediction	SALICON	MDS-ViTNet	CC	0.8980	#4 of 5	Archive leaderboard	report
Saliency Prediction	SALICON	MDS-ViTNet	KLD	0.2127	#4 of 5	Archive leaderboard	report
Saliency Prediction	SALICON	MDS-ViTNet	SIM	0.7887	#4 of 5	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Absolute Position Encodings Adam Attention BPE Dense Connections Dropout Label Smoothing Layer Normalization Linear Layer Multi-Head Attention Position-Wise Feed-Forward Layer Residual Connection Softmax Stochastic Depth Swin Transformer Transformer Vision Transformer

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections