Papers › Vision and Structured-Language Pretraining for Cross-Modal Food Retrieval
Vision and Structured-Language Pretraining for Cross-Modal Food Retrieval
Mustafa Shukor, Nicolas Thome, Matthieu Cord
Vision-Language Pretraining (VLP) and Foundation models have been the go-to recipe for achieving SoTA performance on general benchmarks. However, leveraging these powerful techniques for more complex vision-language tasks, such as cooking applications, with more structured input data, is still little investigated. In this work, we propose to leverage these techniques for structured-text based computational cuisine tasks. Our strategy, dubbed VLPCook, first transforms existing image-text pairs to image and structured-text pairs. This allows to pretrain our VLPCook model using VLP objectives adapted to the strutured data of the resulting datasets, then finetuning it on downstream computational cooking tasks. During finetuning, we also enrich the visual encoder, leveraging pretrained foundation models (e.g. CLIP) to provide local and global textual context. VLPCook outperforms current SoTA by a significant margin (+3.3 Recall@1 absolute improvement) on the task of Cross-Modal Food Retrieval on the large Recipe1M dataset. We conduct further experiments on VLP to validate their importance, especially on the Recipe1M+ dataset. Finally, we validate the generalization of the approach to other tasks (i.e, Food Recognition) and domains with structured text such as the Medical domain on the ROCO dataset. The code is available here: https://github.com/mshukor/VLPCook
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Cross-Modal Retrieval | Recipe1M | VLPCook (R1M+) | Image-to-text R@1 | 74.9 | #1 of 9 | Archive leaderboard | report |
| Cross-Modal Retrieval | Recipe1M | VLPCook (R1M+) | Text-to-image R@1 | 75.6 | #1 of 9 | Archive leaderboard | report |
| Cross-Modal Retrieval | Recipe1M | VLPCook | Image-to-text R@1 | 73.6 | #2 of 9 | Archive leaderboard | report |
| Cross-Modal Retrieval | Recipe1M | VLPCook | Text-to-image R@1 | 74.7 | #2 of 9 | Archive leaderboard | report |
| Cross-Modal Retrieval | Recipe1M+ | VLPCook | Image-to-text R@1 | 45.2 | #1 of 2 | Archive leaderboard | report |
| Cross-Modal Retrieval | Recipe1M+ | VLPCook | Text-to-image R@1 | 47.3 | #1 of 2 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections