Browse State-of-the-Art › Image-to-Text Retrieval
Image-to-Text Retrieval
37 papers with code · 8 benchmarks · 8 datasets archive 2025-07-28
Image-text retrieval is the process of retrieving relevant images based on textual descriptions or finding corresponding textual descriptions for a given image. This task is interdisciplinary, combining techniques from computer vision, and natural language processing. The primary challenge lies in bridging the semantic gap — the difference between how visual data is represented in images and how humans describe that information using language. To address this, many methods focus on learning a shared embedding space where both images and text can be represented in a comparable way, allowing their similarities to be measured and facilitating more accurate retrieval.
Source: Extending CLIP for Category-to-Image Retrieval in E-commerce
Description from the archive archive 2025-07-28.
Benchmarks archive 2025-07-28
8 leaderboard tables shown for this task, 8 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.
Syntology 3 newer papers were judged to report on this task's benchmarks so far (Claude Sonnet 4.5, reading their tables: a model's judgement, not a result; 0 placed): Flickr30k (3 judged, 0 placed), COCO (Common Objects in Context) (1 judged, 0 placed). Drawn from 8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read).
Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
8 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Papers archive 2025-07-28
archive papers tagged: 59 · with a code link: 37 · where Syntology ran a sample: 17 (14 with a run with no instrument failure, 3 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (17 of 59 tagged: 14 with a run with no instrument failure, 3 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 59 of 59, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
26 Feb 2021 82 repositories listed Syntology official: no sample here; runs from other or unrecorded repositories · 16 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 1 violated, 0 with no contract checked; 14 where Syntology's instrument failed) · 4 unverified (of 20 harvested samples) · 16 pointer-only (licence)State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories.
-
30 Jan 2023 17 repositories listed Syntology community repositories only · 4 ran (of which 4 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified; every one of the 4 samples that ran constructed an object rather than computing a result (of 8 harvested samples) · 1 pointer-only (licence)The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models.
-
27 Mar 2023 11 repositories listed Syntology community repositories only · 14 ran (of which 3 constructed an object rather than computing a result; 11 with no instrument failure: 0 honoured, 0 violated, 11 with no contract checked; 3 where Syntology's instrument failed) · 15 unverified (of 29 harvested samples) · 25 pointer-only (licence)We propose a simple pairwise Sigmoid loss for Language-Image Pre-training (SigLIP).
-
16 Jul 2021 6 repositories listed Syntology community repositories only · 4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified (of 5 harvested samples) · 3 pointer-only (licence)Most existing methods employ a transformer-based multimodal encoder to jointly model visual tokens (region-based image features) and word tokens.
-
8 Dec 2021 4 repositories listedState-of-the-art vision and vision-and-language models rely on large-scale visio-linguistic pretraining for obtaining good performance on a variety of downstream tasks.
-
13 Apr 2020 4 repositories listed Syntology official (archive's flag): 1 ran · 13 ran (of which 6 constructed an object rather than computing a result; 10 with no instrument failure: 1 honoured, 0 violated, 9 with no contract checked; 3 where Syntology's instrument failed) · 10 unverified (of 23 harvested samples) · 1 pointer-only (licence)Large-scale pre-training methods of learning cross-modal representations on image-text pairs are becoming popular for vision-language tasks.
-
Our approach leverages datasets of images and their sentence descriptions to learn about the inter-modal correspondences between language and visual data.
-
27 Jan 2022 3 repositories listed Syntology official (archive's flag): 17 ran · 17 ran (of which 2 constructed an object rather than computing a result; 11 with no instrument failure: 2 honoured, 0 violated, 9 with no contract checked; 6 where Syntology's instrument failed) · 4 unverified (of 21 harvested samples) · 3 pointer-only (licence)Our benchmark enables the evaluation of multilingual multimodal models for transfer learning, not only in a zero-shot setting, but also in newly defined few-shot learning setups.
-
21 Dec 2023 2 repositories listed Syntology community repositories only · 2 ran (of which 2 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; every one of the 2 samples that ran constructed an object rather than computing a result (of 2 harvested samples) · 2 pointer-only (licence)However, the progress in vision and vision-language foundation models, which are also critical elements of multi-modal AGI, has not kept pace with LLMs.
-
10 Dec 2023 2 repositories listedSince clean samples are easier distinguished by GMM with increasing noise, the memory bank can still maintain high quality at a high noise ratio.
-
15 Aug 2023 2 repositories listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 1 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)In this work, we design the first vision-language dataset distillation method, building on the idea of trajectory matching.
-
18 May 2023 2 repositories listed Syntology official (archive's flag): 5 ran · 5 ran (of which 2 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 0 violated, 3 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified (of 7 harvested samples) · 1 pointer-only (licence)In this work, we explore a scalable way for building a general representation model toward unlimited modalities.
-
31 Jan 2023 2 repositories listed Syntology official: harvested, nothing ran · 0 ran · 2 unverified (of 2 harvested samples)Real-world data contains a vast amount of multimodal information, among which vision and language are the two most representative modalities.
-
11 Jan 2023 2 repositories listedIt is not easy to propose a new model with a novel architecture and intensively train it on a massive dataset with many GPUs to surpass many SOTA models, which are already available to use on the Internet.
-
6 Dec 2022 2 repositories listedTo verify the effectiveness of our approach, extensive experiments are conducted on MS-COCO, CUB Captions, and Flickr30K, which are commonly used in cross-modal retrieval.
-
12 Nov 2022 2 repositories listed Syntology official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 1 violated, 9 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 11 harvested samples) · 2 pointer-only (licence)In this work, we present a conceptually simple and effective method to train a strong bilingual/multilingual multimodal representation model.
-
1 Jul 2021 2 repositories listedIn this paper, we propose an Omni-perception Pre-Trainer (OPT) for cross-modal understanding and generation, by jointly modeling visual, text and audio resources.
-
11 Mar 2021 2 repositories listedWe further construct a large Chinese multi-source image-text dataset called RUC-CAS-WenLan for pre-training our BriVL model.
-
21 Dec 2017 2 repositories listedFinally, a comprehensive review is presented on the proposed data set to fully advance the task of remote sensing caption.
-
10 Jun 2025 1 repository listedALTA achieves superior performance in vision-language matching tasks like retrieval and zero-shot classification by adapting the pretrained vision model from masked record modeling.
-
30 Jul 2024 1 repository listedWe refer to this bias in associating an activity with the gender of its actual performer in an image or text as the Gender-Activity Binding (GAB) bias and analyze how this bias is internalized in VLMs.
-
10 Jul 2024 1 repository listedHerein, we hypothesize that the pre-trained vision-language models can be utilized for quantitative histopathology image analysis through a simple image-to-text retrieval.
-
14 Jun 2024 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)The novelty of BiVLC is to add a synthetic hard negative image generated from the synthetic text, resulting in two image-to-text retrieval examples (one for each image) and, more importantly, two text-to-image retrieval…
-
28 Apr 2024 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 5 unverified (of 6 harvested samples)We hope this method can facilitate the efficient application of large models to a wide range of downstream tasks while significantly reducing the resource consumption.
-
1 Jan 2024 1 repository listedWe propose a novel Linguistic-Aware Patch Slimming (LAPS) framework for fine-grained alignment which explicitly identifies redundant visual patches with language supervision and rectifies their semantic and spatial…
-
29 Sep 2023 1 repository listed Syntology official (archive's flag): 17 ran · 17 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 5 honoured, 1 violated, 3 with no contract checked; 8 where Syntology's instrument failed) · 2 unverified (of 19 harvested samples) · 8 pointer-only (licence)In this paper, we propose a novel Prototype-based Aleatoric Uncertainty Quantification (PAU) framework to provide trustworthy predictions by quantifying the uncertainty arisen from the inherent data ambiguity.
-
24 Jul 2023 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples)In this paper, we present a prototype representation learning framework incorporating both global and local alignment between medical images and reports.
-
20 Jun 2023 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 7 harvested samples)Moreover, we present an image-text paired dataset in the field of remote sensing (RS), RS5M, which has 5 million RS images with English descriptions.
-
27 May 2023 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified (of 4 harvested samples)To pursue more efficient vision-language Transformers, this paper introduces Cross-Guided Ensemble of Tokens (CrossGET), a general acceleration framework for vision-language Transformers.
-
21 Apr 2023 1 repository listedThe reason is that a large amount of images and texts in the benchmarks are coarse-grained.
-
30 Sep 2022 1 repository listed
-
8 Sep 2022 1 repository listed
-
5 Sep 2022 1 repository listed
-
9 Jun 2022 1 repository listed
-
4 Jun 2021 1 repository listed
-
28 May 2021 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 2 harvested samples)
-
8 Oct 2019 1 repository listed
-
Improving Medical Visual Representation Learning with Pathological-level Cross-Modal Alignment and Correlation Exploration12 Jun 2025 0 repositories listed
-
Towards Cross-modal Retrieval in Chinese Cultural Heritage Documents: Dataset and Solution16 May 2025 0 repositories listed
-
SemCORE: A Semantic-Enhanced Generative Cross-Modal Retrieval Framework with MLLMs17 Apr 2025 0 repositories listed
-
DART: Disease-aware Image-Text Alignment and Self-correcting Re-alignment for Trustworthy Radiology Report Generation16 Apr 2025 0 repositories listed
-
ABC: Achieving Better Control of Multimodal Embeddings using VLMs1 Mar 2025 0 repositories listed
-
Retaining Knowledge and Enhancing Long-Text Representations in CLIP through Dual-Teacher Distillation1 Jan 2025 0 repositories listed
-
DIR: Retrieval-Augmented Image Captioning with Comprehensive Understanding2 Dec 2024 0 repositories listed
-
Robotic State Recognition with Image-to-Text Retrieval Task of Pre-Trained Vision-Language Model and Black-Box Optimization30 Oct 2024 0 repositories listed
-
Robotic Environmental State Recognition with Pre-Trained Vision-Language Models and Black-Box Optimization26 Sep 2024 0 repositories listed
-
Understanding the Effect of using Semantically Meaningful Tokens for Visual Representation Learning26 May 2024 0 repositories listed
-
CLIP the Bias: How Useful is Balancing Data in Multimodal Learning?7 Mar 2024 0 repositories listed
-
Accept the Modality Gap: An Exploration in the Hyperbolic Space1 Jan 2024 0 repositories listed
-
Towards a Visual-Language Foundation Model for Computational Pathology24 Jul 2023 0 repositories listed
-
Is Cross-modal Information Retrieval Possible without Training?20 Apr 2023 0 repositories listed
-
Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images13 Mar 2023 0 repositories listed
-
When are Lemons Purple? The Concept Association Bias of Vision-Language Models22 Dec 2022 0 repositories listed
-
A survey on knowledge-enhanced multimodal learning19 Nov 2022 0 repositories listed
-
Paired Cross-Modal Data Augmentation for Fine-Grained Image-to-Text Retrieval29 Jul 2022 0 repositories listed
-
25 May 2022 0 repositories listed
-
15 Apr 2022 0 repositories listed
-
10 Oct 2020 0 repositories listed
-
16 Aug 2019 0 repositories listed
Syntology lines on 19 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections