{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bridgeformer-bridging-video-text-retrieval","title":"Bridging Video-text Retrieval with Multiple Choice Questions","arxiv_id":"2201.04850","date":"2022-01-13","proceeding":"CVPR 2022 1","authors":["Yuying Ge","Yixiao Ge","Xihui Liu","Dian Li","Ying Shan","XiaoHu Qie","Ping Luo"],"abstract":"Pre-training a model to learn transferable video-text representation for retrieval has attracted a lot of attention in recent years. Previous dominant works mainly adopt two separate encoders for efficient retrieval, but ignore local associations between videos and texts. Another line of research uses a joint encoder to interact video with texts, but results in low efficiency since each text-video pair needs to be fed into the model. In this work, we enable fine-grained video-text interactions while maintaining high efficiency for retrieval via a novel pretext task, dubbed as Multiple Choice Questions (MCQ), where a parametric module BridgeFormer is trained to answer the \"questions\" constructed by the text features via resorting to the video features. Specifically, we exploit the rich semantics of text (i.e., nouns and verbs) to build questions, with which the video encoder can be trained to capture more regional content and temporal dynamics. In the form of questions and answers, the semantic associations between local video-text features can be properly established. BridgeFormer is able to be removed for downstream retrieval, rendering an efficient and flexible model with only two encoders. Our method outperforms state-of-the-art methods on the popular text-to-video retrieval task in five datasets with different experimental setups (i.e., zero-shot and fine-tune), including HowTo100M (one million videos). We further conduct zero-shot action recognition, which can be cast as video-to-text retrieval, and our approach also significantly surpasses its counterparts. As an additional benefit, our method achieves competitive results with much shorter pre-training videos on single-modality downstream tasks, e.g., action recognition with linear evaluation.","url_abs":"https://arxiv.org/abs/2201.04850v2","url_pdf":"https://arxiv.org/pdf/2201.04850v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bridgeformer-bridging-video-text-retrieval","repo_url":"https://github.com/tencentarc/mcq","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"bridgeformer-bridging-video-text-retrieval","repo_url":"https://github.com/towhee-io/towhee","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"linear-evaluation","task_name":"Linear evaluation"},{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"text-retrieval","task_name":"Text Retrieval"},{"task_slug":"text-to-video-retrieval","task_name":"Text to Video Retrieval"},{"task_slug":"text-to-video-search","task_name":"Text-to-video search"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"},{"task_slug":"video-to-text-retrieval","task_name":"Video to Text Retrieval"},{"task_slug":"video-text-retrieval","task_name":"Video-Text Retrieval"},{"task_slug":"zero-shot-action-recognition","task_name":"Zero-Shot Action Recognition"},{"task_slug":"zero-shot-video-retrieval","task_name":"Zero-Shot Video Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-retrieval-on-msr-vtt-1ka","task":"Video Retrieval","dataset":"MSR-VTT-1kA","model":"BridgeFormer","rank_in_archive_order":43,"of":63,"metrics":{"text-to-video Median Rank":"3","text-to-video R@1":"37.6","text-to-video R@10":"75.1","text-to-video R@5":"64.8"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-msr-vtt-1ka","task":"Video Retrieval","dataset":"MSR-VTT-1kA","model":"BridgeFormer (Zero-shot)","rank_in_archive_order":54,"of":63,"metrics":{"text-to-video Median Rank":"7","text-to-video R@1":"26","text-to-video R@10":"56.4","text-to-video R@5":"46.4"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-video-retrieval-on-didemo","task":"Zero-Shot Video Retrieval","dataset":"DiDeMo","model":"Y. Ge et. al.","rank_in_archive_order":19,"of":26,"metrics":{"text-to-video Median Rank":"5.0","text-to-video R@1":"25.6","text-to-video R@10":"61.1","text-to-video R@5":"50.6"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-video-retrieval-on-lsmdc","task":"Zero-Shot Video Retrieval","dataset":"LSMDC","model":"Y. Ge et. al.","rank_in_archive_order":14,"of":16,"metrics":{"text-to-video Median Rank":" 42.0","text-to-video R@1":"12.2","text-to-video R@10":"32.2","text-to-video R@5":"25.9"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-video-retrieval-on-msr-vtt","task":"Zero-Shot Video Retrieval","dataset":"MSR-VTT","model":"Y. Ge et. al.","rank_in_archive_order":27,"of":41,"metrics":{"text-to-video Median Rank":"7.0","text-to-video R@1":"26.0","text-to-video R@10":"56.4","text-to-video R@5":"46.4"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-video-retrieval-on-msvd","task":"Zero-Shot Video Retrieval","dataset":"MSVD","model":"Y. Ge et. al.","rank_in_archive_order":10,"of":14,"metrics":{"text-to-video Median Rank":"2.0","text-to-video R@1":"43.6","text-to-video R@10":"84.9","text-to-video R@5":"74.9"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2201.04850","atlas_url":"https://app.syntology.ai/?focus=2201.04850","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2201.04850"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tencentarc/mcq","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/towhee-io/towhee","reach":null}],"summary":{"ran":9,"ran_draft_wrong":3,"ran_fixture":1,"unverified":11},"by_repo_kind":{"official":{"samples":6,"ran":2,"repositories":1},"listed":{"samples":18,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"cf6f823dcb3ac547","entry":"Conv2d","repo":"towhee-io/towhee","repo_kind":"listed","path":"towhee/models/bridgeformer/bridge_former_training_block.py","file_url":"https://github.com/towhee-io/towhee/blob/HEAD/towhee/models/bridgeformer/bridge_former_training_block.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cf6f823dcb3ac547"}},{"code_sha256_prefix":"697a1de2168a5a59","entry":"CrossAttention","repo":"towhee-io/towhee","repo_kind":"listed","path":"towhee/models/bridgeformer/bridge_former_training_block.py","file_url":"https://github.com/towhee-io/towhee/blob/HEAD/towhee/models/bridgeformer/bridge_former_training_block.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"697a1de2168a5a59"}},{"code_sha256_prefix":"7a61100315c44e2c","entry":"Dropout","repo":"towhee-io/towhee","repo_kind":"listed","path":"towhee/models/bridgeformer/bridge_former_training_block.py","file_url":"https://github.com/towhee-io/towhee/blob/HEAD/towhee/models/bridgeformer/bridge_former_training_block.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7a61100315c44e2c"}},{"code_sha256_prefix":"128875f3ef96bf3c","entry":"Linear","repo":"towhee-io/towhee","repo_kind":"listed","path":"towhee/models/bridgeformer/bridge_former_training_block.py","file_url":"https://github.com/towhee-io/towhee/blob/HEAD/towhee/models/bridgeformer/bridge_former_training_block.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"128875f3ef96bf3c"}},{"code_sha256_prefix":"cbf3c6f7dca35838","entry":"Mlp","repo":"towhee-io/towhee","repo_kind":"listed","path":"towhee/models/bridgeformer/bridge_former_training_block.py","file_url":"https://github.com/towhee-io/towhee/blob/HEAD/towhee/models/bridgeformer/bridge_former_training_block.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cbf3c6f7dca35838"}},{"code_sha256_prefix":"ec2435de7b9ac8aa","entry":"RelProp","repo":"towhee-io/towhee","repo_kind":"listed","path":"towhee/models/bridgeformer/bridge_former_training_block.py","file_url":"https://github.com/towhee-io/towhee/blob/HEAD/towhee/models/bridgeformer/bridge_former_training_block.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ec2435de7b9ac8aa"}},{"code_sha256_prefix":"42c5469c00c98d39","entry":"SpatialTemporalClsPositionalEncoding","repo":"towhee-io/towhee","repo_kind":"listed","path":"towhee/models/bridgeformer/bridge_former_training_block.py","file_url":"https://github.com/towhee-io/towhee/blob/HEAD/towhee/models/bridgeformer/bridge_former_training_block.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42c5469c00c98d39"}},{"code_sha256_prefix":"99880b32f1ae367a","entry":"VarAttention","repo":"towhee-io/towhee","repo_kind":"listed","path":"towhee/models/bridgeformer/bridge_former_training_block.py","file_url":"https://github.com/towhee-io/towhee/blob/HEAD/towhee/models/bridgeformer/bridge_former_training_block.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"99880b32f1ae367a"}},{"code_sha256_prefix":"60f628ac6897aacf","entry":"VideoPatchEmbed","repo":"tencentarc/mcq","repo_kind":"official","path":"model/video_bridge_former.py","file_url":"https://github.com/tencentarc/mcq/blob/HEAD/model/video_bridge_former.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"60f628ac6897aacf"}},{"code_sha256_prefix":"0ef9a056059a14d9","entry":"_ntuple","repo":"towhee-io/towhee","repo_kind":"listed","path":"towhee/models/bridgeformer/bridge_former_training_block.py","file_url":"https://github.com/towhee-io/towhee/blob/HEAD/towhee/models/bridgeformer/bridge_former_training_block.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0ef9a056059a14d9"}},{"code_sha256_prefix":"19f4493dcaecc931","entry":"attn","repo":"towhee-io/towhee","repo_kind":"listed","path":"towhee/models/bridgeformer/bridge_former_training_block.py","file_url":"https://github.com/towhee-io/towhee/blob/HEAD/towhee/models/bridgeformer/bridge_former_training_block.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"19f4493dcaecc931"}},{"code_sha256_prefix":"caf8d3ee2ad34af5","entry":"attn_mask","repo":"towhee-io/towhee","repo_kind":"listed","path":"towhee/models/bridgeformer/bridge_former_training_block.py","file_url":"https://github.com/towhee-io/towhee/blob/HEAD/towhee/models/bridgeformer/bridge_former_training_block.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"caf8d3ee2ad34af5"}},{"code_sha256_prefix":"a3bdf83b03800368","entry":"attn_mask","repo":"tencentarc/mcq","repo_kind":"official","path":"model/video_bridge_former.py","file_url":"https://github.com/tencentarc/mcq/blob/HEAD/model/video_bridge_former.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a3bdf83b03800368"}},{"code_sha256_prefix":"fe726d5b3aa825e3","entry":"CrossAttention","repo":"tencentarc/mcq","repo_kind":"official","path":"model/video_bridge_former.py","file_url":"https://github.com/tencentarc/mcq/blob/HEAD/model/video_bridge_former.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe726d5b3aa825e3"}},{"code_sha256_prefix":"44c7199b24a33141","entry":"GELU","repo":"towhee-io/towhee","repo_kind":"listed","path":"towhee/models/bridgeformer/bridge_former_training_block.py","file_url":"https://github.com/towhee-io/towhee/blob/HEAD/towhee/models/bridgeformer/bridge_former_training_block.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"44c7199b24a33141"}},{"code_sha256_prefix":"830af90036b9557c","entry":"PatchEmbed2D","repo":"towhee-io/towhee","repo_kind":"listed","path":"towhee/models/bridgeformer/bridge_former_training_block.py","file_url":"https://github.com/towhee-io/towhee/blob/HEAD/towhee/models/bridgeformer/bridge_former_training_block.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"830af90036b9557c"}},{"code_sha256_prefix":"e89ee5db86ca051f","entry":"VarAttention","repo":"tencentarc/mcq","repo_kind":"official","path":"model/video_bridge_former.py","file_url":"https://github.com/tencentarc/mcq/blob/HEAD/model/video_bridge_former.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e89ee5db86ca051f"}},{"code_sha256_prefix":"aff6dc9eb36e5314","entry":"VideoBridgeBlock","repo":"towhee-io/towhee","repo_kind":"listed","path":"towhee/models/bridgeformer/bridge_former_training_block.py","file_url":"https://github.com/towhee-io/towhee/blob/HEAD/towhee/models/bridgeformer/bridge_former_training_block.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aff6dc9eb36e5314"}},{"code_sha256_prefix":"395f0de4af2bbeea","entry":"VideoBridgeFormer","repo":"towhee-io/towhee","repo_kind":"listed","path":"towhee/models/bridgeformer/bridge_former_training_block.py","file_url":"https://github.com/towhee-io/towhee/blob/HEAD/towhee/models/bridgeformer/bridge_former_training_block.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"395f0de4af2bbeea"}},{"code_sha256_prefix":"b10786c62626a38c","entry":"Video_Bridge_Block","repo":"tencentarc/mcq","repo_kind":"official","path":"model/video_bridge_former.py","file_url":"https://github.com/tencentarc/mcq/blob/HEAD/model/video_bridge_former.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b10786c62626a38c"}},{"code_sha256_prefix":"7e4f19b9162746fd","entry":"Video_Bridge_Former","repo":"tencentarc/mcq","repo_kind":"official","path":"model/video_bridge_former.py","file_url":"https://github.com/tencentarc/mcq/blob/HEAD/model/video_bridge_former.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7e4f19b9162746fd"}},{"code_sha256_prefix":"5fc693caf332e5ae","entry":"init_vit_weights","repo":"towhee-io/towhee","repo_kind":"listed","path":"towhee/models/bridgeformer/bridge_former_training_block.py","file_url":"https://github.com/towhee-io/towhee/blob/HEAD/towhee/models/bridgeformer/bridge_former_training_block.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5fc693caf332e5ae"}},{"code_sha256_prefix":"d90854b3e1f1b595","entry":"lecun_normal_","repo":"towhee-io/towhee","repo_kind":"listed","path":"towhee/models/bridgeformer/bridge_former_training_block.py","file_url":"https://github.com/towhee-io/towhee/blob/HEAD/towhee/models/bridgeformer/bridge_former_training_block.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d90854b3e1f1b595"}},{"code_sha256_prefix":"f654a0a29887209b","entry":"variance_scaling_","repo":"towhee-io/towhee","repo_kind":"listed","path":"towhee/models/bridgeformer/bridge_former_training_block.py","file_url":"https://github.com/towhee-io/towhee/blob/HEAD/towhee/models/bridgeformer/bridge_former_training_block.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f654a0a29887209b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}