Papers › COTS: Collaborative Two-Stream Vision-Language Pre-Training Model for Cross-Modal Retrieval
COTS: Collaborative Two-Stream Vision-Language Pre-Training Model for Cross-Modal Retrieval
Haoyu Lu, Nanyi Fei, Yuqi Huo, Yizhao Gao, Zhiwu Lu, Ji-Rong Wen
Large-scale single-stream pre-training has shown dramatic performance in image-text retrieval. Regrettably, it faces low inference efficiency due to heavy attention layers. Recently, two-stream methods like CLIP and ALIGN with high inference efficiency have also shown promising performance, however, they only consider instance-level alignment between the two streams (thus there is still room for improvement). To overcome these limitations, we propose a novel COllaborative Two-Stream vision-language pretraining model termed COTS for image-text retrieval by enhancing cross-modal interaction. In addition to instance level alignment via momentum contrastive learning, we leverage two extra levels of cross-modal interactions in our COTS: (1) Token-level interaction - a masked visionlanguage modeling (MVLM) learning objective is devised without using a cross-stream network module, where variational autoencoder is imposed on the visual encoder to generate visual tokens for each image. (2) Task-level interaction - a KL-alignment learning objective is devised between text-to-image and image-to-text retrieval tasks, where the probability distribution per task is computed with the negative queues in momentum contrastive learning. Under a fair comparison setting, our COTS achieves the highest performance among all two-stream methods and comparable performance (but with 10,800X faster in inference) w.r.t. the latest single-stream methods. Importantly, our COTS is also applicable to text-to-video retrieval, yielding new state-ofthe-art on the widely-used MSR-VTT dataset.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
No code repository is listed for this paper in the archive or in Syntology's graph.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Video Retrieval | MSR-VTT | COTS | text-to-video Median Rank | 3 | #24 of 40 | Archive leaderboard | report |
| Video Retrieval | MSR-VTT | COTS | text-to-video R@1 | 32.1 | #24 of 40 | Archive leaderboard | report |
| Video Retrieval | MSR-VTT | COTS | text-to-video R@10 | 70.2 | #24 of 40 | Archive leaderboard | report |
| Video Retrieval | MSR-VTT | COTS | text-to-video R@5 | 60.8 | #24 of 40 | Archive leaderboard | report |
| Video Retrieval | MSR-VTT-1kA | COTS | text-to-video Median Rank | 2 | #45 of 63 | Archive leaderboard | report |
| Video Retrieval | MSR-VTT-1kA | COTS | text-to-video R@1 | 36.8 | #45 of 63 | Archive leaderboard | report |
| Video Retrieval | MSR-VTT-1kA | COTS | text-to-video R@10 | 73.2 | #45 of 63 | Archive leaderboard | report |
| Video Retrieval | MSR-VTT-1kA | COTS | text-to-video R@5 | 63.8 | #45 of 63 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections