{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fine-tuned-clip-models-are-efficient-video","title":"Fine-tuned CLIP Models are Efficient Video Learners","arxiv_id":"2212.03640","date":"2022-12-06","proceeding":"CVPR 2023 1","authors":["Hanoona Rasheed","Muhammad Uzair Khattak","Muhammad Maaz","Salman Khan","Fahad Shahbaz Khan"],"abstract":"Large-scale multi-modal training with image-text pairs imparts strong generalization to CLIP model. Since training on a similar scale for videos is infeasible, recent approaches focus on the effective transfer of image-based CLIP to the video domain. In this pursuit, new parametric modules are added to learn temporal information and inter-frame relationships which require meticulous design efforts. Furthermore, when the resulting models are learned on videos, they tend to overfit on the given task distribution and lack in generalization aspect. This begs the following question: How to effectively transfer image-level CLIP representations to videos? In this work, we show that a simple Video Fine-tuned CLIP (ViFi-CLIP) baseline is generally sufficient to bridge the domain gap from images to videos. Our qualitative analysis illustrates that the frame-level processing from CLIP image-encoder followed by feature pooling and similarity matching with corresponding text embeddings helps in implicitly modeling the temporal cues within ViFi-CLIP. Such fine-tuning helps the model to focus on scene dynamics, moving objects and inter-object relationships. For low-data regimes where full fine-tuning is not viable, we propose a `bridge and prompt' approach that first uses fine-tuning to bridge the domain gap and then learns prompts on language and vision side to adapt CLIP representations. We extensively evaluate this simple yet strong baseline on zero-shot, base-to-novel generalization, few-shot and fully supervised settings across five video benchmarks. Our code is available at https://github.com/muzairkhattak/ViFi-CLIP.","url_abs":"https://arxiv.org/abs/2212.03640v3","url_pdf":"https://arxiv.org/pdf/2212.03640v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fine-tuned-clip-models-are-efficient-video","repo_url":"https://github.com/muzairkhattak/vifi-clip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2212.03640","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2212.03640"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/muzairkhattak/vifi-clip","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":4,"ran":3,"unverified":10},"by_repo_kind":{"official":{"samples":17,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"98f385d847636a3e","entry":"basic_clean","repo":"muzairkhattak/vifi-clip","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/muzairkhattak/vifi-clip/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"98f385d847636a3e"}},{"code_sha256_prefix":"d919ae32e5e4e616","entry":"get_pairs","repo":"muzairkhattak/vifi-clip","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/muzairkhattak/vifi-clip/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d919ae32e5e4e616"}},{"code_sha256_prefix":"4728dc2025227bba","entry":"one_hot","repo":"muzairkhattak/vifi-clip","repo_kind":"official","path":"datasets/blending.py","file_url":"https://github.com/muzairkhattak/vifi-clip/blob/HEAD/datasets/blending.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4728dc2025227bba"}},{"code_sha256_prefix":"e866e1196986f728","entry":"shear_x","repo":"muzairkhattak/vifi-clip","repo_kind":"official","path":"datasets/rand_augment.py","file_url":"https://github.com/muzairkhattak/vifi-clip/blob/HEAD/datasets/rand_augment.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e866e1196986f728"}},{"code_sha256_prefix":"7b2503c99c9447fb","entry":"shear_y","repo":"muzairkhattak/vifi-clip","repo_kind":"official","path":"datasets/rand_augment.py","file_url":"https://github.com/muzairkhattak/vifi-clip/blob/HEAD/datasets/rand_augment.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b2503c99c9447fb"}},{"code_sha256_prefix":"323759f906fbb394","entry":"translate_x_rel","repo":"muzairkhattak/vifi-clip","repo_kind":"official","path":"datasets/rand_augment.py","file_url":"https://github.com/muzairkhattak/vifi-clip/blob/HEAD/datasets/rand_augment.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"323759f906fbb394"}},{"code_sha256_prefix":"9542161e9640b858","entry":"whitespace_clean","repo":"muzairkhattak/vifi-clip","repo_kind":"official","path":"clip/simple_tokenizer.py","file_url":"https://github.com/muzairkhattak/vifi-clip/blob/HEAD/clip/simple_tokenizer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9542161e9640b858"}},{"code_sha256_prefix":"0c32459bef92ff8c","entry":"auto_resume_helper","repo":"muzairkhattak/vifi-clip","repo_kind":"official","path":"utils/tools.py","file_url":"https://github.com/muzairkhattak/vifi-clip/blob/HEAD/utils/tools.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c32459bef92ff8c"}},{"code_sha256_prefix":"585280ed7d96fed0","entry":"build_model","repo":"muzairkhattak/vifi-clip","repo_kind":"official","path":"clip/model.py","file_url":"https://github.com/muzairkhattak/vifi-clip/blob/HEAD/clip/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"585280ed7d96fed0"}},{"code_sha256_prefix":"c5c15e35688f86b3","entry":"build_optimizer","repo":"muzairkhattak/vifi-clip","repo_kind":"official","path":"utils/optimizer.py","file_url":"https://github.com/muzairkhattak/vifi-clip/blob/HEAD/utils/optimizer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c5c15e35688f86b3"}},{"code_sha256_prefix":"b0a5beb34716d5a6","entry":"check_keywords_in_name","repo":"muzairkhattak/vifi-clip","repo_kind":"official","path":"utils/optimizer.py","file_url":"https://github.com/muzairkhattak/vifi-clip/blob/HEAD/utils/optimizer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b0a5beb34716d5a6"}},{"code_sha256_prefix":"80450600f7f09b0a","entry":"create_logger","repo":"muzairkhattak/vifi-clip","repo_kind":"official","path":"utils/logger.py","file_url":"https://github.com/muzairkhattak/vifi-clip/blob/HEAD/utils/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"80450600f7f09b0a"}},{"code_sha256_prefix":"f9d64435c7cf1ada","entry":"get_config","repo":"muzairkhattak/vifi-clip","repo_kind":"official","path":"utils/config.py","file_url":"https://github.com/muzairkhattak/vifi-clip/blob/HEAD/utils/config.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9d64435c7cf1ada"}},{"code_sha256_prefix":"0d93231b83e79763","entry":"load","repo":"muzairkhattak/vifi-clip","repo_kind":"official","path":"clip/clip.py","file_url":"https://github.com/muzairkhattak/vifi-clip/blob/HEAD/clip/clip.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d93231b83e79763"}},{"code_sha256_prefix":"9743cf8efee35995","entry":"load_checkpoint","repo":"muzairkhattak/vifi-clip","repo_kind":"official","path":"utils/tools.py","file_url":"https://github.com/muzairkhattak/vifi-clip/blob/HEAD/utils/tools.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9743cf8efee35995"}},{"code_sha256_prefix":"2e6dd851990fe932","entry":"reduce_tensor","repo":"muzairkhattak/vifi-clip","repo_kind":"official","path":"utils/tools.py","file_url":"https://github.com/muzairkhattak/vifi-clip/blob/HEAD/utils/tools.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2e6dd851990fe932"}},{"code_sha256_prefix":"883e1ab4ad47ea2d","entry":"set_weight_decay","repo":"muzairkhattak/vifi-clip","repo_kind":"official","path":"utils/optimizer.py","file_url":"https://github.com/muzairkhattak/vifi-clip/blob/HEAD/utils/optimizer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"883e1ab4ad47ea2d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}