Zero-Shot Cross-Modal Retrieval

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

3 leaderboard tables shown for this task, 3 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

5 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

No subtask under this task in the archive's task tree.

Parent tasks archive 2025-07-28

Most implemented papers archive 2025-07-28

22 shown of 22 papers with code (26 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• Learning Transferable Visual Models From Natural Language Supervision
  26 Feb 2021 82 repositories listed Syntology ran 16 of 20 samples · 4 unverified · 16 pointer-only (licence)
  State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories.
• UNITER: UNiversal Image-TExt Representation Learning
  25 Sep 2019 7 repositories listed Syntology ran 3 of 3 samples · 0 unverified · 2 pointer-only (licence)
  Different from previous work that applies joint random masking to both modalities, we use conditional masking on pre-training tasks (i.
• CoCa: Contrastive Captioners are Image-Text Foundation Models
  4 May 2022 6 repositories listed Syntology ran 9 of 17 samples · 8 unverified
  We apply a contrastive loss between unimodal image and text embeddings, in addition to a captioning loss on the multimodal decoder outputs which predicts text tokens autoregressively.
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
  16 Jul 2021 6 repositories listed Syntology ran 3 of 5 samples · 2 unverified · 3 pointer-only (licence)
  Most existing methods employ a transformer-based multimodal encoder to jointly model visual tokens (region-based image features) and word tokens.
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
  5 Feb 2021 6 repositories listed Syntology ran 1 of 4 samples · 3 unverified · 1 pointer-only (licence)
  Vision-and-Language Pre-training (VLP) has improved performance on various joint vision-and-language downstream tasks.
• Reproducible scaling laws for contrastive language-image learning
  14 Dec 2022 5 repositories listed Syntology ran 3 of 3 samples · 0 unverified · 3 pointer-only (licence)
  To address these limitations, we investigate scaling laws for contrastive language-image pre-training (CLIP) with the public LAION dataset and the open-source OpenCLIP repository.
• Flamingo: a Visual Language Model for Few-Shot Learning
  29 Apr 2022 5 repositories listed Syntology ran 18 of 24 samples · 6 unverified · 7 pointer-only (licence)
  Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research.
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
  11 Feb 2021 5 repositories listed Syntology ran 8 of 10 samples · 2 unverified · 9 pointer-only (licence)
  In this paper, we leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps in the Conceptual Captions dataset.
• Merlin: A Vision Language Foundation Model for 3D Computed Tomography
  10 Jun 2024 2 repositories listed Syntology ran 1 of 9 samples · 8 unverified
  However, current medical VLMs are generally limited to 2D images and short reports, and do not leverage electronic health record (EHR) data for supervision.
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
  21 Dec 2023 2 repositories listed Syntology ran 2 of 2 samples · 0 unverified · 2 pointer-only (licence)
  However, the progress in vision and vision-language foundation models, which are also critical elements of multi-modal AGI, has not kept pace with LLMs.
• VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset
  29 May 2023 2 repositories listed Syntology ran 15 of 42 samples · 27 unverified
  Based on the proposed VAST-27M dataset, we train an omni-modality video-text foundational model named VAST, which can perceive and process vision, audio, and subtitle modalities from video, and better support various…
• Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers
  11 May 2023 2 repositories listed Syntology ran 5 of 8 samples · 3 unverified
  We present Region-aware Open-vocabulary Vision Transformers (RO-ViT) - a contrastive image-text pretraining recipe to bridge the gap between image-level pretraining and open-vocabulary object detection.
• AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities
  12 Nov 2022 2 repositories listed Syntology ran 2 of 11 samples · 9 unverified
  In this work, we present a conceptually simple and effective method to train a strong bilingual/multilingual multimodal representation model.
• Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
  22 Aug 2022 2 repositories listed
  A big convergence of language, vision, and multimodal pretraining is emerging.
• Florence: A New Foundation Model for Computer Vision
  22 Nov 2021 2 repositories listed
  Computer vision foundation models, which are trained on diverse, large-scale dataset and can be adapted to a wide range of downstream tasks, are critical for this mission to solve real-world computer vision applications.
• IMPACT: A Large-scale Integrated Multimodal Patent Analysis and Creation Dataset for Design Patents
  10 Dec 2024 1 repository listed
  In this paper, we introduce IMPACT (Integrated Multimodal Patent Analysis and Creation Dataset for Design Patents), a large-scale multimodal patent dataset with detailed captions for design patent figures.
• COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training
  2 Dec 2024 1 repository listed
  Vision-Language Models (VLMs) trained with contrastive loss have achieved significant advancements in various vision and language tasks.
• M2-Encoder: Advancing Bilingual Image-Text Understanding by Large-scale Efficient Pretraining
  29 Jan 2024 1 repository listed
  Vision-language foundation models like CLIP have revolutionized the field of artificial intelligence.
• Implicit Differentiable Outlier Detection Enable Robust Deep Multimodal Analysis
  21 Sep 2023 1 repository listed
  Deep network models are often purely inductive during both training and inference on unseen data.
• Position-guided Text Prompt for Vision-Language Pre-training
  19 Dec 2022 1 repository listed Syntology ran 3 of 5 samples · 2 unverified
  In this work, we propose a novel Position-guided Text Prompt (PTP) paradigm to enhance the visual grounding ability of cross-modal models trained with VLP.
• ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-training
  30 Sep 2022 1 repository listed
  They attempt to learn cross-modal representation using contrastive learning on image-text pairs, however, the built inter-modal correlations only rely on a single view for each modality.
• Vision-Language Pre-Training with Triple Contrastive Learning
  21 Feb 2022 1 repository listed Syntology ran 1 of 2 samples · 1 unverified
  Besides CMA, TCL introduces an intra-modal contrastive objective to provide complementary benefits in representation learning.

Syntology lines on 15 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.