{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/diffusionret-generative-text-video-retrieval","title":"DiffusionRet: Generative Text-Video Retrieval with Diffusion Model","arxiv_id":"2303.09867","date":"2023-03-17","proceeding":"ICCV 2023 1","authors":["Peng Jin","Hao Li","Zesen Cheng","Kehan Li","Xiangyang Ji","Chang Liu","Li Yuan","Jie Chen"],"abstract":"Existing text-video retrieval solutions are, in essence, discriminant models focused on maximizing the conditional likelihood, i.e., p(candidates|query). While straightforward, this de facto paradigm overlooks the underlying data distribution p(query), which makes it challenging to identify out-of-distribution data. To address this limitation, we creatively tackle this task from a generative viewpoint and model the correlation between the text and the video as their joint probability p(candidates,query). This is accomplished through a diffusion-based text-video retrieval framework (DiffusionRet), which models the retrieval task as a process of gradually generating joint distribution from noise. During training, DiffusionRet is optimized from both the generation and discrimination perspectives, with the generator being optimized by generation loss and the feature extractor trained with contrastive loss. In this way, DiffusionRet cleverly leverages the strengths of both generative and discriminative methods. Extensive experiments on five commonly used text-video retrieval benchmarks, including MSRVTT, LSMDC, MSVD, ActivityNet Captions, and DiDeMo, with superior performances, justify the efficacy of our method. More encouragingly, without any modification, DiffusionRet even performs well in out-domain retrieval settings. We believe this work brings fundamental insights into the related fields. Code is available at https://github.com/jpthu17/DiffusionRet.","url_abs":"https://arxiv.org/abs/2303.09867v2","url_pdf":"https://arxiv.org/pdf/2303.09867v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"diffusionret-generative-text-video-retrieval","repo_url":"https://github.com/jpthu17/diffusionret","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"diffusionret-generative-text-video-retrieval","repo_url":"https://github.com/jpthu17/HBI","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"diffusionret-generative-text-video-retrieval","repo_url":"https://github.com/jpthu17/dicosa","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"diffusionret-generative-text-video-retrieval","repo_url":"https://github.com/jpthu17/emcl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-retrieval-on-activitynet","task":"Video Retrieval","dataset":"ActivityNet","model":"DiffusionRet+QB-Norm","rank_in_archive_order":18,"of":31,"metrics":{"text-to-video Mean Rank":"6.8","text-to-video Median Rank":"2.0","text-to-video R@1":"48.1","text-to-video R@10":"85.7","video-to-text Mean Rank":"6.7","video-to-text Median Rank":"2.0","video-to-text R@1":"47.4","video-to-text R@10":"86.7","video-to-text R@5":"76.3"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-activitynet","task":"Video Retrieval","dataset":"ActivityNet","model":"DiffusionRet","rank_in_archive_order":22,"of":31,"metrics":{"text-to-video Mean Rank":"6.5","text-to-video Median Rank":"2.0","text-to-video R@1":"45.8","text-to-video R@10":"86.3","text-to-video R@5":"75.6","video-to-text Mean Rank":"6.3","video-to-text Median Rank":"2.0","video-to-text R@1":"43.8","video-to-text R@10":"86.7","video-to-text R@5":"75.3"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-didemo","task":"Video Retrieval","dataset":"DiDeMo","model":"DiffusionRet+QB-Norm","rank_in_archive_order":26,"of":40,"metrics":{"text-to-video Mean Rank":"14.1","text-to-video Median Rank":"2.0","text-to-video R@1":"48.9","text-to-video R@10":"83.3","text-to-video R@5":"75.5","video-to-text Mean Rank":"10.3","video-to-text Median Rank":"1.0","video-to-text R@1":"50.3","video-to-text R@10":"82.9","video-to-text R@5":"75.1"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-didemo","task":"Video Retrieval","dataset":"DiDeMo","model":"DiffusionRet","rank_in_archive_order":31,"of":40,"metrics":{"text-to-video Mean Rank":"14.3","text-to-video Median Rank":"2.0","text-to-video R@1":"46.7","text-to-video R@10":"82.7","text-to-video R@5":"74.7","video-to-text Mean Rank":"10.7","video-to-text Median Rank":"2.0","video-to-text R@1":"46.2","video-to-text R@10":"82.2","video-to-text R@5":"74.3"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-lsmdc","task":"Video Retrieval","dataset":"LSMDC","model":"DiffusionRet","rank_in_archive_order":19,"of":38,"metrics":{"text-to-video Mean Rank":"40.7","text-to-video Median Rank":"8.0","text-to-video R@1":"24.4","text-to-video R@10":"54.3","text-to-video R@5":"43.1","video-to-text Mean Rank":"40.2","video-to-text Median Rank":"9.0","video-to-text R@1":"23.0","video-to-text R@10":"51.5","video-to-text R@5":"43.5"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-msr-vtt-1ka","task":"Video Retrieval","dataset":"MSR-VTT-1kA","model":"DiffusionRet","rank_in_archive_order":21,"of":63,"metrics":{"text-to-video Mean Rank":"12.1","text-to-video Median Rank":"2.0","text-to-video R@1":"49.0","text-to-video R@10":"82.7","text-to-video R@5":"75.2","video-to-text Mean Rank":"8.8","video-to-text Median Rank":"2.0","video-to-text R@1":"47.7","video-to-text R@10":"84.5","video-to-text R@5":"73.8"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-msr-vtt-1ka","task":"Video Retrieval","dataset":"MSR-VTT-1kA","model":"DiffusionRet+QB-Norm","rank_in_archive_order":22,"of":63,"metrics":{"text-to-video Mean Rank":"12.1","text-to-video Median Rank":"2.0","text-to-video R@1":"48.9","text-to-video R@10":"83.1","text-to-video R@5":"75.2","video-to-text Mean Rank":"8.5","video-to-text Median Rank":"2.0","video-to-text R@1":"49.3","video-to-text R@10":"83.8","video-to-text R@5":"74.3"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-msvd","task":"Video Retrieval","dataset":"MSVD","model":"DiffusionRet+QB-Norm","rank_in_archive_order":15,"of":24,"metrics":{"text-to-video Mean Rank":"15.6","text-to-video R@1":"47.9","text-to-video R@10":"84.8","text-to-video R@5":"77.2","video-to-text Mean Rank":"4.5","video-to-text Median Rank":"1.0","video-to-text R@1":"60.3","video-to-text R@10":"92","video-to-text R@5":"86.4"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-msvd","task":"Video Retrieval","dataset":"MSVD","model":"DiffusionRet","rank_in_archive_order":18,"of":24,"metrics":{"text-to-video Mean Rank":"15.7","text-to-video Median Rank":"2.0","text-to-video R@1":"46.6","text-to-video R@10":"84.1","text-to-video R@5":"75.9","video-to-text Mean Rank":"4.5","video-to-text Median Rank":"1.0","video-to-text R@1":"61.9","video-to-text R@10":"92.9","video-to-text R@5":"88.3"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2303.09867","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.09867"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jpthu17/dicosa","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jpthu17/HBI","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jpthu17/emcl","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jpthu17/diffusionret","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/jpthu17/DiffusionRet","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":5,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a8984912b201c0e2","entry":"ArcCrossEn","repo":"jpthu17/DiffusionRet","repo_kind":"official","path":"DiffusionRet/models/modeling.py","file_url":"https://github.com/jpthu17/DiffusionRet/blob/HEAD/DiffusionRet/models/modeling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a8984912b201c0e2"}},{"code_sha256_prefix":"862a183cbe47818a","entry":"Diffusion","repo":"jpthu17/DiffusionRet","repo_kind":"official","path":"DiffusionRet/models/modeling.py","file_url":"https://github.com/jpthu17/DiffusionRet/blob/HEAD/DiffusionRet/models/modeling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"862a183cbe47818a"}},{"code_sha256_prefix":"54d61676ebbe2fec","entry":"Diffusion_v","repo":"jpthu17/DiffusionRet","repo_kind":"official","path":"DiffusionRet/models/modeling.py","file_url":"https://github.com/jpthu17/DiffusionRet/blob/HEAD/DiffusionRet/models/modeling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"54d61676ebbe2fec"}},{"code_sha256_prefix":"2c961a941bf02a05","entry":"PositionalEncoding","repo":"jpthu17/DiffusionRet","repo_kind":"official","path":"DiffusionRet/models/modeling.py","file_url":"https://github.com/jpthu17/DiffusionRet/blob/HEAD/DiffusionRet/models/modeling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2c961a941bf02a05"}},{"code_sha256_prefix":"29b441f9817457ba","entry":"TimestepEmbedder","repo":"jpthu17/DiffusionRet","repo_kind":"official","path":"DiffusionRet/models/modeling.py","file_url":"https://github.com/jpthu17/DiffusionRet/blob/HEAD/DiffusionRet/models/modeling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"29b441f9817457ba"}},{"code_sha256_prefix":"6963fb4ce6b8de6c","entry":"DiffusionRet","repo":"jpthu17/DiffusionRet","repo_kind":"official","path":"DiffusionRet/models/modeling.py","file_url":"https://github.com/jpthu17/DiffusionRet/blob/HEAD/DiffusionRet/models/modeling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6963fb4ce6b8de6c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}