{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/use-what-you-have-video-retrieval-using","title":"Use What You Have: Video Retrieval Using Representations From Collaborative Experts","arxiv_id":"1907.13487","date":"2019-07-31","proceeding":null,"authors":["Yang Liu","Samuel Albanie","Arsha Nagrani","Andrew Zisserman"],"abstract":"The rapid growth of video on the internet has made searching for video content using natural language queries a significant challenge. Human-generated queries for video datasets `in the wild' vary a lot in terms of degree of specificity, with some queries describing specific details such as the names of famous identities, content from speech, or text available on the screen. Our goal is to condense the multi-modal, extremely high dimensional information from videos into a single, compact video representation for the task of video retrieval using free-form text queries, where the degree of specificity is open-ended. For this we exploit existing knowledge in the form of pre-trained semantic embeddings which include 'general' features such as motion, appearance, and scene features from visual content. We also explore the use of more 'specific' cues from ASR and OCR which are intermittently available for videos and find that these signals remain challenging to use effectively for retrieval. We propose a collaborative experts model to aggregate information from these different pre-trained experts and assess our approach empirically on five retrieval benchmarks: MSR-VTT, LSMDC, MSVD, DiDeMo, and ActivityNet. Code and data can be found at www.robots.ox.ac.uk/~vgg/research/collaborative-experts/. This paper contains a correction to results reported in the previous version.","url_abs":"https://arxiv.org/abs/1907.13487v2","url_pdf":"https://arxiv.org/pdf/1907.13487v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"use-what-you-have-video-retrieval-using","repo_url":"https://github.com/albanie/collaborative-experts","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"use-what-you-have-video-retrieval-using","repo_url":"https://github.com/caijincen712/CE","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"use-what-you-have-video-retrieval-using","repo_url":"https://github.com/hbzhang/cvpr2020","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"natural-language-queries","task_name":"Natural Language Queries"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"specificity","task_name":"Specificity"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-retrieval-on-activitynet","task":"Video Retrieval","dataset":"ActivityNet","model":"Collaborative Experts","rank_in_archive_order":31,"of":31,"metrics":{"text-to-video Mean Rank":"23.1","text-to-video Median Rank":"6","text-to-video R@1":"20.5","text-to-video R@10":"63.9","text-to-video R@5":"47.7","text-to-video R@50":"91.4"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-didemo","task":"Video Retrieval","dataset":"DiDeMo","model":"Collaborative Experts","rank_in_archive_order":39,"of":40,"metrics":{"text-to-video Mean Rank":"43.7","text-to-video Median Rank":"8.3","text-to-video R@1":"16.1","text-to-video R@10":"54.4","text-to-video R@5":"41.1","text-to-video R@50":"82.7"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-lsmdc","task":"Video Retrieval","dataset":"LSMDC","model":"Collaborative Experts","rank_in_archive_order":32,"of":38,"metrics":{"text-to-video Median Rank":"25","text-to-video R@1":"11.2","text-to-video R@10":"34.8","text-to-video R@5":"26.9"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-msr-vtt","task":"Video Retrieval","dataset":"MSR-VTT","model":"Collaborative Experts","rank_in_archive_order":37,"of":40,"metrics":{"text-to-video Mean Rank":"86.8","text-to-video Median Rank":"16","text-to-video R@1":"10.0","text-to-video R@10":"41.2","text-to-video R@5":"29.0","video-to-text Mean Rank":"38.1","video-to-text Median Rank":"8.3","video-to-text R@1":"15.6","video-to-text R@10":"55.2","video-to-text R@5":"40.9"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-msr-vtt-1ka","task":"Video Retrieval","dataset":"MSR-VTT-1kA","model":"Collaborative Experts","rank_in_archive_order":56,"of":63,"metrics":{"text-to-video Mean Rank":"28.2","text-to-video Median Rank":"6","text-to-video R@1":"20.9","text-to-video R@10":"62.4","text-to-video R@5":"48.8"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-msvd","task":"Video Retrieval","dataset":"MSVD","model":"Collaborative Experts","rank_in_archive_order":24,"of":24,"metrics":{"text-to-video Mean Rank":"23.1","text-to-video Median Rank":"6.0","text-to-video R@1":"19.8","text-to-video R@10":"63.8","text-to-video R@5":"49.0","text-to-video R@50":"89.0"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1907.13487","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1907.13487"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/albanie/collaborative-experts","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hbzhang/cvpr2020","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/caijincen712/CE","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":5},"by_repo_kind":{"listed":{"samples":5,"ran":0,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"fc6aa51656b4be1e","entry":"kronecker_prod","repo":"hbzhang/cvpr2020","repo_kind":"listed","path":"model/model.py","file_url":"https://github.com/hbzhang/cvpr2020/blob/HEAD/model/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fc6aa51656b4be1e"}},{"code_sha256_prefix":"7ccb106acb20a26e","entry":"retrieval_as_classification","repo":"caijincen712/CE","repo_kind":"listed","path":"model/metric.py","file_url":"https://github.com/caijincen712/CE/blob/HEAD/model/metric.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7ccb106acb20a26e"}},{"code_sha256_prefix":"ba4c695148436f05","entry":"sent_feat","repo":"caijincen712/CE","repo_kind":"listed","path":"sent_feat_demo.py","file_url":"https://github.com/caijincen712/CE/blob/HEAD/sent_feat_demo.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ba4c695148436f05"}},{"code_sha256_prefix":"7f4ebeefd49432b9","entry":"t2v_metrics","repo":"caijincen712/CE","repo_kind":"listed","path":"model/metric.py","file_url":"https://github.com/caijincen712/CE/blob/HEAD/model/metric.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7f4ebeefd49432b9"}},{"code_sha256_prefix":"0ce3d614411abce7","entry":"v2t_metrics","repo":"caijincen712/CE","repo_kind":"listed","path":"model/metric.py","file_url":"https://github.com/caijincen712/CE/blob/HEAD/model/metric.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0ce3d614411abce7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}