{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/covr-learning-composed-video-retrieval-from","title":"CoVR-2: Automatic Data Construction for Composed Video Retrieval","arxiv_id":"2308.14746","date":"2023-08-28","proceeding":null,"authors":["Lucas Ventura","Antoine Yang","Cordelia Schmid","Gül Varol"],"abstract":"Composed Image Retrieval (CoIR) has recently gained popularity as a task that considers both text and image queries together, to search for relevant images in a database. Most CoIR approaches require manually annotated datasets, comprising image-text-image triplets, where the text describes a modification from the query image to the target image. However, manual curation of CoIR triplets is expensive and prevents scalability. In this work, we instead propose a scalable automatic dataset creation methodology that generates triplets given video-caption pairs, while also expanding the scope of the task to include composed video retrieval (CoVR). To this end, we mine paired videos with a similar caption from a large database, and leverage a large language model to generate the corresponding modification text. Applying this methodology to the extensive WebVid2M collection, we automatically construct our WebVid-CoVR dataset, resulting in 1.6 million triplets. Moreover, we introduce a new benchmark for CoVR with a manually annotated evaluation set, along with baseline results. We further validate that our methodology is equally applicable to image-caption pairs, by generating 3.3 million CoIR training triplets using the Conceptual Captions dataset. Our model builds on BLIP-2 pretraining, adapting it to composed video (or image) retrieval, and incorporates an additional caption retrieval loss to exploit extra supervision beyond the triplet. We provide extensive ablations to analyze the design choices on our new CoVR benchmark. Our experiments also demonstrate that training a CoVR model on our datasets effectively transfers to CoIR, leading to improved state-of-the-art performance in the zero-shot setup on the CIRR, FashionIQ, and CIRCO benchmarks. Our code, datasets, and models are publicly available at https://imagine.enpc.fr/ ventural/covr.","url_abs":"https://arxiv.org/abs/2308.14746v4","url_pdf":"https://arxiv.org/pdf/2308.14746v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"covr-learning-composed-video-retrieval-from","repo_url":"https://github.com/lucas-ventura/CoVR","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"composed-image-retrieval","task_name":"Composed Image Retrieval (CoIR)"},{"task_slug":"composed-video-retrieval-covr","task_name":"Composed Video Retrieval (CoVR)"},{"task_slug":"image-retrieval","task_name":"Image Retrieval"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":null,"task_name":"Triplet"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"},{"task_slug":"zero-shot-composed-image-retrieval-zs-cir","task_name":"Zero-Shot Composed Image Retrieval (ZS-CIR)"}],"methods":[{"method_slug":"covr","method_name":"CoVR"}],"datasets_introduced":[{"slug":"webvid-covr","name":"WebVid-CoVR","full_name":""}],"methods_introduced":[{"slug":"covr","name":"CoVR","full_name":"Composed Video Retrieval"}],"results":[{"leaderboard":"/sota/composed-image-retrieval-coir-on-cirr-1","task":"Composed Image Retrieval (CoIR)","dataset":"CIRR","model":"CoVR-BLIP-2","rank_in_archive_order":1,"of":1,"metrics":{"R@1":"50.43","R@5":"81.08"},"uses_additional_data":false},{"leaderboard":"/sota/composed-image-retrieval-coir-on-fashion-iq","task":"Composed Image Retrieval (CoIR)","dataset":"Fashion IQ","model":"CoVR-BLIP-2","rank_in_archive_order":1,"of":1,"metrics":{"(Recall@10+Recall@50)/2":"60.57","R@10":"49.96","R@50":"71.17"},"uses_additional_data":false},{"leaderboard":"/sota/composed-video-retrieval-covr-on-covr","task":"Composed Video Retrieval (CoVR)","dataset":"WebVid-CoVR","model":"BLIP-2","rank_in_archive_order":1,"of":1,"metrics":{"R@1":"59.82"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRCO","model":"CoVR-BLIP-2","rank_in_archive_order":13,"of":43,"metrics":{"mAP@10":"29.55"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-1","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"CIRR","model":"CoVR-BLIP-2","rank_in_archive_order":2,"of":47,"metrics":{"R@1":"43.74","R@10":"83.95","R@5":"73.61","R@50":"96.1"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-composed-image-retrieval-zs-cir-on-2","task":"Zero-Shot Composed Image Retrieval (ZS-CIR)","dataset":"Fashion IQ","model":"CoVR-BLIP-2","rank_in_archive_order":6,"of":41,"metrics":{"(Recall@10+Recall@50)/2":"48.3","R@10":"38.15","R@50":"58.44"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2308.14746","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.14746"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lucas-ventura/CoVR","reach":null}],"summary":{"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0c4f22b80e11b371","entry":"get_blip_config","repo":"lucas-ventura/CoVR","repo_kind":"official","path":"tools/embs/save_blip_embs_vids.py","file_url":"https://github.com/lucas-ventura/CoVR/blob/HEAD/tools/embs/save_blip_embs_vids.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c4f22b80e11b371"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}