{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/frozen-clip-models-are-efficient-video","title":"Frozen CLIP Models are Efficient Video Learners","arxiv_id":"2208.03550","date":"2022-08-06","proceeding":null,"authors":["Ziyi Lin","Shijie Geng","Renrui Zhang","Peng Gao","Gerard de Melo","Xiaogang Wang","Jifeng Dai","Yu Qiao","Hongsheng Li"],"abstract":"Video recognition has been dominated by the end-to-end learning paradigm -- first initializing a video recognition model with weights of a pretrained image model and then conducting end-to-end training on videos. This enables the video network to benefit from the pretrained image model. However, this requires substantial computation and memory resources for finetuning on videos and the alternative of directly using pretrained image features without finetuning the image backbone leads to subpar results. Fortunately, recent advances in Contrastive Vision-Language Pre-training (CLIP) pave the way for a new route for visual recognition tasks. Pretrained on large open-vocabulary image-text pair data, these models learn powerful visual representations with rich semantics. In this paper, we present Efficient Video Learning (EVL) -- an efficient framework for directly training high-quality video recognition models with frozen CLIP features. Specifically, we employ a lightweight Transformer decoder and learn a query token to dynamically collect frame-level spatial features from the CLIP image encoder. Furthermore, we adopt a local temporal module in each decoder layer to discover temporal clues from adjacent frames and their attention maps. We show that despite being efficient to train with a frozen backbone, our models learn high quality video representations on a variety of video recognition datasets. Code is available at https://github.com/OpenGVLab/efficient-video-recognition.","url_abs":"https://arxiv.org/abs/2208.03550v1","url_pdf":"https://arxiv.org/pdf/2208.03550v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"frozen-clip-models-are-efficient-video","repo_url":"https://github.com/opengvlab/efficient-video-recognition","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"frozen-clip-models-are-efficient-video","repo_url":"https://github.com/chenhsing/svformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"action-classification","task_name":"Action Classification"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"video-recognition","task_name":"Video Recognition"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"EVL (CLIP ViT-L/14@336px, frozen, 32 frames)","rank_in_archive_order":32,"of":207,"metrics":{"Acc@1":"87.7","Acc@5":"97.8"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2208.03550","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2208.03550"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/opengvlab/efficient-video-recognition","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chenhsing/svformer","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/OpenGVLab/efficient-video-recognition","reach":{"status":"ok"}}],"summary":{"ran":5,"unverified":5},"by_repo_kind":{"official":{"samples":10,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"68359211beb5845f","entry":"Attention","repo":"OpenGVLab/efficient-video-recognition","repo_kind":"official","path":"model.py","file_url":"https://github.com/OpenGVLab/efficient-video-recognition/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"68359211beb5845f"}},{"code_sha256_prefix":"6dc8dfed11a95191","entry":"PatchEmbed2D","repo":"OpenGVLab/efficient-video-recognition","repo_kind":"official","path":"model.py","file_url":"https://github.com/OpenGVLab/efficient-video-recognition/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6dc8dfed11a95191"}},{"code_sha256_prefix":"939b58dc96e7688d","entry":"TemporalCrossAttention","repo":"OpenGVLab/efficient-video-recognition","repo_kind":"official","path":"model.py","file_url":"https://github.com/OpenGVLab/efficient-video-recognition/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"939b58dc96e7688d"}},{"code_sha256_prefix":"672cd6b662676846","entry":"TransformerDecoderLayer","repo":"OpenGVLab/efficient-video-recognition","repo_kind":"official","path":"model.py","file_url":"https://github.com/OpenGVLab/efficient-video-recognition/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"672cd6b662676846"}},{"code_sha256_prefix":"827c460499de0c70","entry":"TransformerEncoderLayer","repo":"OpenGVLab/efficient-video-recognition","repo_kind":"official","path":"model.py","file_url":"https://github.com/OpenGVLab/efficient-video-recognition/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"827c460499de0c70"}},{"code_sha256_prefix":"ebbaaa44f01bf0b0","entry":"EVLDecoder","repo":"OpenGVLab/efficient-video-recognition","repo_kind":"official","path":"model.py","file_url":"https://github.com/OpenGVLab/efficient-video-recognition/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ebbaaa44f01bf0b0"}},{"code_sha256_prefix":"5cfcbd5ab2e4afb2","entry":"EVLTransformer","repo":"OpenGVLab/efficient-video-recognition","repo_kind":"official","path":"model.py","file_url":"https://github.com/OpenGVLab/efficient-video-recognition/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5cfcbd5ab2e4afb2"}},{"code_sha256_prefix":"da39da2dd6bd01ff","entry":"VisionTransformer2D","repo":"OpenGVLab/efficient-video-recognition","repo_kind":"official","path":"model.py","file_url":"https://github.com/OpenGVLab/efficient-video-recognition/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"da39da2dd6bd01ff"}},{"code_sha256_prefix":"ee488f9d9228c231","entry":"load_weights_clip","repo":"OpenGVLab/efficient-video-recognition","repo_kind":"official","path":"model.py","file_url":"https://github.com/OpenGVLab/efficient-video-recognition/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ee488f9d9228c231"}},{"code_sha256_prefix":"4a8075639cafecc1","entry":"model_to_fp16","repo":"OpenGVLab/efficient-video-recognition","repo_kind":"official","path":"model.py","file_url":"https://github.com/OpenGVLab/efficient-video-recognition/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4a8075639cafecc1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}