{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/x-pool-cross-modal-language-video-attention","title":"X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrieval","arxiv_id":"2203.15086","date":"2022-03-28","proceeding":"CVPR 2022 1","authors":["Satya Krishna Gorti","Noel Vouitsis","Junwei Ma","Keyvan Golestan","Maksims Volkovs","Animesh Garg","Guangwei Yu"],"abstract":"In text-video retrieval, the objective is to learn a cross-modal similarity function between a text and a video that ranks relevant text-video pairs higher than irrelevant pairs. However, videos inherently express a much wider gamut of information than texts. Instead, texts often capture sub-regions of entire videos and are most semantically similar to certain frames within videos. Therefore, for a given text, a retrieval model should focus on the text's most semantically similar video sub-regions to make a more relevant comparison. Yet, most existing works aggregate entire videos without directly considering text. Common text-agnostic aggregations schemes include mean-pooling or self-attention over the frames, but these are likely to encode misleading visual information not described in the given text. To address this, we propose a cross-modal attention model called X-Pool that reasons between a text and the frames of a video. Our core mechanism is a scaled dot product attention for a text to attend to its most semantically similar frames. We then generate an aggregated video representation conditioned on the text's attention weights over the frames. We evaluate our method on three benchmark datasets of MSR-VTT, MSVD and LSMDC, achieving new state-of-the-art results by up to 12% in relative improvement in Recall@1. Our findings thereby highlight the importance of joint text-video reasoning to extract important visual cues according to text. Full code and demo can be found at: https://layer6ai-labs.github.io/xpool/","url_abs":"https://arxiv.org/abs/2203.15086v1","url_pdf":"https://arxiv.org/pdf/2203.15086v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"x-pool-cross-modal-language-video-attention","repo_url":"https://github.com/layer6ai-labs/xpool","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"text-to-video-retrieval","task_name":"Text to Video Retrieval"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"},{"task_slug":"video-text-retrieval","task_name":"Video-Text Retrieval"}],"methods":[{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-retrieval-on-lsmdc","task":"Video Retrieval","dataset":"LSMDC","model":"X-Pool","rank_in_archive_order":17,"of":38,"metrics":{"text-to-video Mean Rank":"53.2","text-to-video Median Rank":"8.0","text-to-video R@1":"25.2","text-to-video R@10":"53.5","text-to-video R@5":"43.7","video-to-text Mean Rank":"47.4","video-to-text Median Rank":"10.0","video-to-text R@1":"22.7","video-to-text R@10":"51.2","video-to-text R@5":"42.6"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-msr-vtt-1ka","task":"Video Retrieval","dataset":"MSR-VTT-1kA","model":"X-Pool","rank_in_archive_order":30,"of":63,"metrics":{"text-to-video Mean Rank":"14.3","text-to-video Median Rank":"2","text-to-video R@1":"46.9","text-to-video R@10":"82.2","text-to-video R@5":"72.8","video-to-text Mean Rank":"9.0","video-to-text Median Rank":"2.0","video-to-text R@1":"44.4","video-to-text R@10":"84.0","video-to-text R@5":"73.3"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-msvd","task":"Video Retrieval","dataset":"MSVD","model":"X-Pool","rank_in_archive_order":17,"of":24,"metrics":{"text-to-video Mean Rank":"9.3","text-to-video Median Rank":"2.0","text-to-video R@1":"47.2","text-to-video R@10":"86.0","text-to-video R@5":"77.4","video-to-text Mean Rank":"3.3","video-to-text Median Rank":"1.0","video-to-text R@1":"66.4","video-to-text R@10":"94.2","video-to-text R@5":"90.0"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2203.15086","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.15086"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/layer6ai-labs/xpool","reach":null}],"summary":{"ran":1,"unverified":1},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"f082c8f319e8ce4a","entry":"BaselinePooling","repo":"layer6ai-labs/xpool","repo_kind":"official","path":"modules/baseline_pooling.py","file_url":"https://github.com/layer6ai-labs/xpool/blob/HEAD/modules/baseline_pooling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f082c8f319e8ce4a"}},{"code_sha256_prefix":"f0930dba90bc5cd5","entry":"Config","repo":"layer6ai-labs/xpool","repo_kind":"official","path":"modules/baseline_pooling.py","file_url":"https://github.com/layer6ai-labs/xpool/blob/HEAD/modules/baseline_pooling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f0930dba90bc5cd5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}