{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/prompt-switch-efficient-clip-adaptation-for","title":"Prompt Switch: Efficient CLIP Adaptation for Text-Video Retrieval","arxiv_id":"2308.07648","date":"2023-08-15","proceeding":"ICCV 2023 1","authors":["Chaorui Deng","Qi Chen","Pengda Qin","Da Chen","Qi Wu"],"abstract":"In text-video retrieval, recent works have benefited from the powerful learning capabilities of pre-trained text-image foundation models (e.g., CLIP) by adapting them to the video domain. A critical problem for them is how to effectively capture the rich semantics inside the video using the image encoder of CLIP. To tackle this, state-of-the-art methods adopt complex cross-modal modeling techniques to fuse the text information into video frame representations, which, however, incurs severe efficiency issues in large-scale retrieval systems as the video representations must be recomputed online for every text query. In this paper, we discard this problematic cross-modal fusion process and aim to learn semantically-enhanced representations purely from the video, so that the video representations can be computed offline and reused for different texts. Concretely, we first introduce a spatial-temporal \"Prompt Cube\" into the CLIP image encoder and iteratively switch it within the encoder layers to efficiently incorporate the global video semantics into frame representations. We then propose to apply an auxiliary video captioning objective to train the frame representations, which facilitates the learning of detailed video semantics by providing fine-grained guidance in the semantic space. With a naive temporal fusion strategy (i.e., mean-pooling) on the enhanced frame representations, we obtain state-of-the-art performances on three benchmark datasets, i.e., MSR-VTT, MSVD, and LSMDC.","url_abs":"https://arxiv.org/abs/2308.07648v1","url_pdf":"https://arxiv.org/pdf/2308.07648v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"prompt-switch-efficient-clip-adaptation-for","repo_url":"https://github.com/bladewaltz1/promptswitch","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"video-captioning","task_name":"Video Captioning"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2308.07648","atlas_url":"https://app.syntology.ai/?focus=2308.07648","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.07648"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bladewaltz1/promptswitch","reach":null}],"summary":{"ran":3,"ran_draft_wrong":1,"unverified":12},"by_repo_kind":{"official":{"samples":16,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":16,"samples":[{"code_sha256_prefix":"6b5c6a4c5fe3629a","entry":"CLIPEncoderLayer","repo":"bladewaltz1/promptswitch","repo_kind":"official","path":"model/prompt_clip.py","file_url":"https://github.com/bladewaltz1/promptswitch/blob/HEAD/model/prompt_clip.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6b5c6a4c5fe3629a"}},{"code_sha256_prefix":"7afff4a1c7789597","entry":"CLIPTextEmbeddings","repo":"bladewaltz1/promptswitch","repo_kind":"official","path":"model/prompt_clip.py","file_url":"https://github.com/bladewaltz1/promptswitch/blob/HEAD/model/prompt_clip.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7afff4a1c7789597"}},{"code_sha256_prefix":"97b7de8e8d588a6e","entry":"CLIPVisionEmbeddings","repo":"bladewaltz1/promptswitch","repo_kind":"official","path":"model/prompt_clip.py","file_url":"https://github.com/bladewaltz1/promptswitch/blob/HEAD/model/prompt_clip.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"97b7de8e8d588a6e"}},{"code_sha256_prefix":"749dd08d5d7409f2","entry":"_expand_mask","repo":"bladewaltz1/promptswitch","repo_kind":"official","path":"model/prompt_clip.py","file_url":"https://github.com/bladewaltz1/promptswitch/blob/HEAD/model/prompt_clip.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"749dd08d5d7409f2"}},{"code_sha256_prefix":"6b28b5c3f3f4e210","entry":"BaselinePooling","repo":"bladewaltz1/promptswitch","repo_kind":"official","path":"model/prompt_clip.py","file_url":"https://github.com/bladewaltz1/promptswitch/blob/HEAD/model/prompt_clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6b28b5c3f3f4e210"}},{"code_sha256_prefix":"4c0e317bbad36a96","entry":"CLIPAttention","repo":"bladewaltz1/promptswitch","repo_kind":"official","path":"model/prompt_clip.py","file_url":"https://github.com/bladewaltz1/promptswitch/blob/HEAD/model/prompt_clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4c0e317bbad36a96"}},{"code_sha256_prefix":"4521a46931a2dd05","entry":"CLIPEncoder","repo":"bladewaltz1/promptswitch","repo_kind":"official","path":"model/prompt_clip.py","file_url":"https://github.com/bladewaltz1/promptswitch/blob/HEAD/model/prompt_clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4521a46931a2dd05"}},{"code_sha256_prefix":"072dd8bae2230fa6","entry":"CLIPEncoderLayerPT","repo":"bladewaltz1/promptswitch","repo_kind":"official","path":"model/prompt_clip.py","file_url":"https://github.com/bladewaltz1/promptswitch/blob/HEAD/model/prompt_clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"072dd8bae2230fa6"}},{"code_sha256_prefix":"01439931fe5d4f12","entry":"CLIPEncoderPT","repo":"bladewaltz1/promptswitch","repo_kind":"official","path":"model/prompt_clip.py","file_url":"https://github.com/bladewaltz1/promptswitch/blob/HEAD/model/prompt_clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"01439931fe5d4f12"}},{"code_sha256_prefix":"90804502b77c9f49","entry":"CLIPModel","repo":"bladewaltz1/promptswitch","repo_kind":"official","path":"model/prompt_clip.py","file_url":"https://github.com/bladewaltz1/promptswitch/blob/HEAD/model/prompt_clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"90804502b77c9f49"}},{"code_sha256_prefix":"be95c1088bdc81ad","entry":"CLIPTextTransformer","repo":"bladewaltz1/promptswitch","repo_kind":"official","path":"model/prompt_clip.py","file_url":"https://github.com/bladewaltz1/promptswitch/blob/HEAD/model/prompt_clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"be95c1088bdc81ad"}},{"code_sha256_prefix":"925873e2ceea0336","entry":"CLIPVisionTransformer","repo":"bladewaltz1/promptswitch","repo_kind":"official","path":"model/prompt_clip.py","file_url":"https://github.com/bladewaltz1/promptswitch/blob/HEAD/model/prompt_clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"925873e2ceea0336"}},{"code_sha256_prefix":"e28877d426856a6f","entry":"Config","repo":"bladewaltz1/promptswitch","repo_kind":"official","path":"model/prompt_clip.py","file_url":"https://github.com/bladewaltz1/promptswitch/blob/HEAD/model/prompt_clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e28877d426856a6f"}},{"code_sha256_prefix":"90011c4f1d727ebe","entry":"MultiHeadedAttention","repo":"bladewaltz1/promptswitch","repo_kind":"official","path":"model/prompt_clip.py","file_url":"https://github.com/bladewaltz1/promptswitch/blob/HEAD/model/prompt_clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"90011c4f1d727ebe"}},{"code_sha256_prefix":"87939011e0a0b2dd","entry":"PromptCLIP","repo":"bladewaltz1/promptswitch","repo_kind":"official","path":"model/prompt_clip.py","file_url":"https://github.com/bladewaltz1/promptswitch/blob/HEAD/model/prompt_clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"87939011e0a0b2dd"}},{"code_sha256_prefix":"3654adee70b77587","entry":"Transformer","repo":"bladewaltz1/promptswitch","repo_kind":"official","path":"model/prompt_clip.py","file_url":"https://github.com/bladewaltz1/promptswitch/blob/HEAD/model/prompt_clip.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3654adee70b77587"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}