Papers › WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning

WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning

2 Mar 2021arXiv:2103.01913archive 2025-07-28

Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, Marc Najork

The milestone improvements brought about by deep representation learning and pre-training techniques have led to large performance gains across downstream NLP, IR and Vision tasks. Multimodal modeling techniques aim to leverage large high-quality visio-linguistic datasets for learning complementary information (across image and text modalities). In this paper, we introduce the Wikipedia-based Image Text (WIT) Dataset (https://github.com/google-research-datasets/wit) to better facilitate multimodal, multilingual learning. WIT is composed of a curated set of 37.6 million entity rich image-text examples with 11.5 million unique images across 108 Wikipedia languages. Its size enables WIT to be used as a pretraining dataset for multimodal models, as we show when applied to downstream tasks such as image-text retrieval. WIT has four main and unique advantages. First, WIT is the largest multimodal dataset by the number of image-text examples by 3x (at the time of writing). Second, WIT is massively multilingual (first of its kind) with coverage over 100+ languages (each of which has at least 12K examples) and provides cross-lingual texts for many images. Third, WIT represents a more diverse set of concepts and real world entities relative to what previous datasets cover. Lastly, WIT provides a very challenging real-world test set, as we empirically illustrate using an image-text retrieval task as an example.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

google-research-datasets/wit officialmentioned in papermentioned on GitHubNOASSERTIONSyntology: not harvested report
clip-italian/clip-italian mentioned on GitHubjaxSyntology: not harvested report
paullerner/viquae mentioned on GitHubpytorchNOASSERTIONSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Cross-modal Retrieval for Knowledge-based Visual Question Answering (arXiv:2401.05736; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 7 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

BIG-bench Machine LearningImage RetrievalImage-text RetrievalRepresentation LearningRetrievalText Retrieval

Datasets

Introduced by this paper, per the archive.

WIT

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Image Retrieval WIT WIT-ALL R@1 0.346 #1 of 2 Archive leaderboard report
Image Retrieval WIT WIT-ALL R@5 0.642 #1 of 2 Archive leaderboard report
Image Retrieval WIT CC (Conceptual Captions) R@1 0.048 #2 of 2 Archive leaderboard report
Image Retrieval WIT CC (Conceptual Captions) R@5 0.122 #2 of 2 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections