Cross-Modal Retrieval

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

13 leaderboard tables shown for this task, 13 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted. 10 shown of 13 until expanded.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

24 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

5 subtasks in the archive's task tree.

Parent tasks archive 2025-07-28

Most implemented papers archive 2025-07-28

30 shown of 244 papers with code (522 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• Stacked Capsule Autoencoders
  17 Jun 2019 11 repositories listed Syntology ran 2 of 29 samples · 27 unverified
  In the second stage, SCAE predicts parameters of a few object capsules, which are then used to reconstruct part poses.
• VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
  18 Jul 2017 10 repositories listed Syntology ran 2 of 5 samples · 3 unverified · 4 pointer-only (licence)
  We present a new technique for learning visual-semantic embeddings for cross-modal retrieval.
• Rescaling Egocentric Vision
  23 Jun 2020 7 repositories listed Syntology ran 3 of 18 samples · 15 unverified
  This paper introduces the pipeline to extend the largest dataset in egocentric vision, EPIC-KITCHENS.
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
  16 Jul 2021 6 repositories listed Syntology ran 3 of 5 samples · 2 unverified · 3 pointer-only (licence)
  Most existing methods employ a transformer-based multimodal encoder to jointly model visual tokens (region-based image features) and word tokens.
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
  5 Feb 2021 6 repositories listed Syntology ran 1 of 4 samples · 3 unverified · 1 pointer-only (licence)
  Vision-and-Language Pre-training (VLP) has improved performance on various joint vision-and-language downstream tasks.
• Stacked Cross Attention for Image-Text Matching
  21 Mar 2018 6 repositories listed Syntology ran 7 of 16 samples · 9 unverified · 1 pointer-only (licence)
  Prior work either simply aggregates the similarity of all possible pairs of regions and words without attending differentially to more and less important words or regions, or uses a multi-step attentional process to…
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
  11 Feb 2021 5 repositories listed Syntology ran 8 of 10 samples · 2 unverified · 9 pointer-only (licence)
  In this paper, we leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps in the Conceptual Captions dataset.
• A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language
  12 Sep 2022 4 repositories listed Syntology ran 2 of 8 samples · 6 unverified · 2 pointer-only (licence)
  Although artificial intelligence (AI) has made significant progress in understanding molecules in a wide range of fields, existing models generally acquire the single cognitive ability from the single molecular modality.
• Probabilistic Embeddings for Cross-Modal Retrieval
  13 Jan 2021 4 repositories listed Syntology ran 3 of 4 samples · 1 unverified
  Instead, we propose to use Probabilistic Cross-Modal Embedding (PCME), where samples from the different modalities are represented as probabilistic distributions in the common embedding space.
• Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
  13 Apr 2020 4 repositories listed Syntology ran 11 of 23 samples · 12 unverified
  Large-scale pre-training methods of learning cross-modal representations on image-text pairs are becoming popular for vision-language tasks.
• Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoning
  1 Mar 2020 4 repositories listed Syntology ran 0 of 7 samples · 7 unverified
  To improve fine-grained video-text retrieval, we propose a Hierarchical Graph Reasoning (HGR) model, which decomposes video-text matching into global-to-local levels.
• Deep Visual-Semantic Alignments for Generating Image Descriptions
  7 Dec 2014 4 repositories listed Syntology ran 0 of 1 samples · 1 unverified
  Our approach leverages datasets of images and their sentence descriptions to learn about the inter-modal correspondences between language and visual data.
• ImageBind: One Embedding Space To Bind Them All
  9 May 2023 3 repositories listed Syntology ran 24 of 34 samples · 10 unverified · 32 pointer-only (licence)
  We show that all combinations of paired data are not necessary to train such a joint embedding, and only image-paired data is sufficient to bind the modalities together.
• A Channel Mix Method for Fine-Grained Cross-Modal Retrieval
  26 Aug 2022 3 repositories listed
  In this paper, we propose a simple but effective method for dealing with the challenging fine-grained cross-modal retrieval task where it aims to enable flexible retrieval among subor-dinate categories across different…
• IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages
  27 Jan 2022 3 repositories listed Syntology ran 8 of 21 samples · 13 unverified
  Our benchmark enables the evaluation of multilingual multimodal models for transfer learning, not only in a zero-shot setting, but also in newly defined few-shot learning setups.
• An Empirical Study of Training End-to-End Vision-and-Language Transformers
  3 Nov 2021 3 repositories listed Syntology ran 1 of 1 samples · 0 unverified
  Vision-and-language (VL) pre-training has proven to be highly effective on various VL downstream tasks.
• UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning
  31 Dec 2020 3 repositories listed
  Existed pre-training methods either focus on single-modal tasks or multi-modal tasks, and cannot effectively adapt to each other.
• FashionBERT: Text and Image Matching with Adaptive Loss for Cross-modal Retrieval
  20 May 2020 3 repositories listed
  In this paper, we address the text and image matching in cross-modal retrieval of the fashion industry.
• Derm1M: A Million-scale Vision-Language Dataset Aligned with Clinical Ontology Knowledge for Dermatology
  19 Mar 2025 2 repositories listed
  However, progress in dermatology has lagged behind other medical domains due to the lack of standard image-text pairs.
• Multimodal Whole Slide Foundation Model for Pathology
  29 Nov 2024 2 repositories listed Syntology ran 2 of 3 samples · 1 unverified · 3 pointer-only (licence)
  The field of computational pathology has been transformed with recent advances in foundation models that encode histopathology region-of-interests (ROIs) into versatile and transferable feature representations via…
• Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation
  30 Sep 2024 2 repositories listed Syntology ran 9 of 15 samples · 6 unverified · 15 pointer-only (licence)
  Surgical video-language pretraining (VLP) faces unique challenges due to the knowledge domain gap and the scarcity of multi-modal data.
• Merlin: A Vision Language Foundation Model for 3D Computed Tomography
  10 Jun 2024 2 repositories listed Syntology ran 1 of 9 samples · 8 unverified
  However, current medical VLMs are generally limited to 2D images and short reports, and do not leverage electronic health record (EHR) data for supervision.
• LeanVec: Searching vectors faster by making them fit
  26 Dec 2023 2 repositories listed
  In this work, we present LeanVec, a framework that combines linear dimensionality reduction with vector quantization to accelerate similarity search on high-dimensional vectors while maintaining accuracy.
• Quilt-1M: One Million Image-Text Pairs for Histopathology
  20 Jun 2023 2 repositories listed Syntology ran 1 of 1 samples · 0 unverified
  From YouTube, we curate QUILT: a large-scale vision-language dataset consisting of $802, 144$ image and text pairs.
• Reducing Semantic Confusion: Scene-aware Aggregation Network for Remote Sensing Cross-modal Retrieval
  12 Jun 2023 2 repositories listed
  Furthermore, as the diversity and differentiation of remote sensing scenes weaken the understanding of scenes, a new metric, namely, scene recall is proposed to measure the perception of scenes by evaluating scene-level…
• MolFM: A Multimodal Molecular Foundation Model
  6 Jun 2023 2 repositories listed
  In this study, we introduce MolFM, a multimodal molecular foundation model designed to facilitate joint representation learning from molecular structures, biomedical texts, and knowledge graphs.
• VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset
  29 May 2023 2 repositories listed Syntology ran 15 of 42 samples · 27 unverified
  Based on the proposed VAST-27M dataset, we train an omni-modality video-text foundational model named VAST, which can perceive and process vision, audio, and subtitle modalities from video, and better support various…
• AToMiC: An Image/Text Retrieval Test Collection to Support Multimedia Content Creation
  4 Apr 2023 2 repositories listed
  This paper presents the AToMiC (Authoring Tools for Multimedia Content) dataset, designed to advance research in image/text cross-modal retrieval.
• Semantic-Conditional Diffusion Networks for Image Captioning
  6 Dec 2022 2 repositories listed
  The rich semantics are further regarded as semantic prior to trigger the learning of Diffusion Transformer, which produces the output sentence in a diffusion process.
• A Differentiable Semantic Metric Approximation in Probabilistic Embedding for Cross-Modal Retrieval
  6 Dec 2022 2 repositories listed
  To verify the effectiveness of our approach, extensive experiments are conducted on MS-COCO, CUB Captions, and Flickr30K, which are commonly used in cross-modal retrieval.

Syntology lines on 20 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.