{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/expanding-language-image-pretrained-models","title":"Expanding Language-Image Pretrained Models for General Video Recognition","arxiv_id":"2208.02816","date":"2022-08-04","proceeding":null,"authors":["Bolin Ni","Houwen Peng","Minghao Chen","Songyang Zhang","Gaofeng Meng","Jianlong Fu","Shiming Xiang","Haibin Ling"],"abstract":"Contrastive language-image pretraining has shown great success in learning visual-textual joint representation from web-scale data, demonstrating remarkable \"zero-shot\" generalization ability for various image tasks. However, how to effectively expand such new language-image pretraining methods to video domains is still an open problem. In this work, we present a simple yet effective approach that adapts the pretrained language-image models to video recognition directly, instead of pretraining a new model from scratch. More concretely, to capture the long-range dependencies of frames along the temporal dimension, we propose a cross-frame attention mechanism that explicitly exchanges information across frames. Such module is lightweight and can be plugged into pretrained language-image models seamlessly. Moreover, we propose a video-specific prompting scheme, which leverages video content information for generating discriminative textual prompts. Extensive experiments demonstrate that our approach is effective and can be generalized to different video recognition scenarios. In particular, under fully-supervised settings, our approach achieves a top-1 accuracy of 87.1% on Kinectics-400, while using 12 times fewer FLOPs compared with Swin-L and ViViT-H. In zero-shot experiments, our approach surpasses the current state-of-the-art methods by +7.6% and +14.9% in terms of top-1 accuracy under two popular protocols. In few-shot scenarios, our approach outperforms previous best methods by +32.1% and +23.1% when the labeled data is extremely limited. Code and models are available at https://aka.ms/X-CLIP","url_abs":"https://arxiv.org/abs/2208.02816v1","url_pdf":"https://arxiv.org/pdf/2208.02816v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"expanding-language-image-pretrained-models","repo_url":"https://github.com/microsoft/videox","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"expanding-language-image-pretrained-models","repo_url":"https://github.com/microsoft/VideoX/tree/master/X-CLIP","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"action-classification","task_name":"Action Classification"},{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"video-recognition","task_name":"Video Recognition"},{"task_slug":"zero-shot-action-recognition","task_name":"Zero-Shot Action Recognition"},{"task_slug":"zero-shot-generalization","task_name":"Zero-shot Generalization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"X-CLIP(ViT-L/14, CLIP)","rank_in_archive_order":34,"of":207,"metrics":{"Acc@1":"87.7","Acc@5":"97.4"},"uses_additional_data":false},{"leaderboard":"/sota/action-classification-on-kinetics-600","task":"Action Classification","dataset":"Kinetics-600","model":"X-CLIP(ViT-L/14, CLIP)","rank_in_archive_order":21,"of":65,"metrics":{"Top-1 Accuracy":"88.3","Top-5 Accuracy":"97.7"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-action-recognition-on-hmdb51","task":"Zero-Shot Action Recognition","dataset":"HMDB51","model":"X-CLIP","rank_in_archive_order":13,"of":29,"metrics":{"Top-1 Accuracy":"44.6"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-action-recognition-on-kinetics","task":"Zero-Shot Action Recognition","dataset":"Kinetics","model":"X-CLIP","rank_in_archive_order":9,"of":20,"metrics":{"Top-1 Accuracy":"65.2","Top-5 Accuracy":"86.1"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-action-recognition-on-ucf101","task":"Zero-Shot Action Recognition","dataset":"UCF101","model":"X-CLIP","rank_in_archive_order":15,"of":35,"metrics":{"Top-1 Accuracy":"72.0"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2208.02816","atlas_url":"https://app.syntology.ai/?focus=2208.02816","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2208.02816"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/VideoX/tree/master/X-CLIP","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/videox","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"24d72b3dd24e3dbe","entry":"MulitHeadAttention","repo":"microsoft/videox","repo_kind":"official","path":"X-CLIP/models/prompt.py","file_url":"https://github.com/microsoft/videox/blob/HEAD/X-CLIP/models/prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"24d72b3dd24e3dbe"}},{"code_sha256_prefix":"d6bc7452b4eff554","entry":"PromptGeneratorLayer","repo":"microsoft/videox","repo_kind":"official","path":"X-CLIP/models/prompt.py","file_url":"https://github.com/microsoft/videox/blob/HEAD/X-CLIP/models/prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d6bc7452b4eff554"}},{"code_sha256_prefix":"5fddf59ff99f898b","entry":"VideoSpecificPrompt","repo":"microsoft/videox","repo_kind":"official","path":"X-CLIP/models/prompt.py","file_url":"https://github.com/microsoft/videox/blob/HEAD/X-CLIP/models/prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5fddf59ff99f898b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}