{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bidirectional-cross-modal-knowledge","title":"Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Models","arxiv_id":"2301.00182","date":"2022-12-31","proceeding":"CVPR 2023 1","authors":["Wenhao Wu","Xiaohan Wang","Haipeng Luo","Jingdong Wang","Yi Yang","Wanli Ouyang"],"abstract":"Vision-language models (VLMs) pre-trained on large-scale image-text pairs have demonstrated impressive transferability on various visual tasks. Transferring knowledge from such powerful VLMs is a promising direction for building effective video recognition models. However, current exploration in this field is still limited. We believe that the greatest value of pre-trained VLMs lies in building a bridge between visual and textual domains. In this paper, we propose a novel framework called BIKE, which utilizes the cross-modal bridge to explore bidirectional knowledge: i) We introduce the Video Attribute Association mechanism, which leverages the Video-to-Text knowledge to generate textual auxiliary attributes for complementing video recognition. ii) We also present a Temporal Concept Spotting mechanism that uses the Text-to-Video expertise to capture temporal saliency in a parameter-free manner, leading to enhanced video representation. Extensive studies on six popular video datasets, including Kinetics-400 & 600, UCF-101, HMDB-51, ActivityNet and Charades, show that our method achieves state-of-the-art performance in various recognition scenarios, such as general, zero-shot, and few-shot video recognition. Our best model achieves a state-of-the-art accuracy of 88.6% on the challenging Kinetics-400 using the released CLIP model. The code is available at https://github.com/whwu95/BIKE .","url_abs":"https://arxiv.org/abs/2301.00182v2","url_pdf":"https://arxiv.org/pdf/2301.00182v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bidirectional-cross-modal-knowledge","repo_url":"https://github.com/whwu95/BIKE","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"bidirectional-cross-modal-knowledge","repo_url":"https://github.com/whwu95/ATM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"bidirectional-cross-modal-knowledge","repo_url":"https://github.com/whwu95/Cap4Video","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"bidirectional-cross-modal-knowledge","repo_url":"https://github.com/whwu95/GPT4Vis","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"bidirectional-cross-modal-knowledge","repo_url":"https://github.com/whwu95/text4vis","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"action-classification","task_name":"Action Classification"},{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"video-recognition","task_name":"Video Recognition"},{"task_slug":"zero-shot-action-recognition","task_name":"Zero-Shot Action Recognition"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-classification-on-charades","task":"Action Classification","dataset":"Charades","model":"BIKE","rank_in_archive_order":10,"of":49,"metrics":{"MAP":"50.7"},"uses_additional_data":false},{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"BIKE (CLIP ViT-L/14)","rank_in_archive_order":21,"of":207,"metrics":{"Acc@1":"88.7","Acc@5":"98.4"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-activitynet","task":"Action Recognition","dataset":"ActivityNet","model":"BIKE","rank_in_archive_order":2,"of":16,"metrics":{"mAP":"96.1"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-hmdb-51","task":"Action Recognition","dataset":"HMDB-51","model":"BIKE","rank_in_archive_order":13,"of":77,"metrics":{"Average accuracy of 3 splits":"83.1"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-in-videos-on-ucf101","task":"Action Recognition","dataset":"UCF101","model":"BIKE","rank_in_archive_order":5,"of":91,"metrics":{"3-fold Accuracy":"98.8"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-action-recognition-on-activitynet","task":"Zero-Shot Action Recognition","dataset":"ActivityNet","model":"BIKE","rank_in_archive_order":1,"of":5,"metrics":{"Top-1 Accuracy":"86.2"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-action-recognition-on-hmdb51","task":"Zero-Shot Action Recognition","dataset":"HMDB51","model":"BIKE","rank_in_archive_order":3,"of":29,"metrics":{"Top-1 Accuracy":"61.4"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-action-recognition-on-kinetics","task":"Zero-Shot Action Recognition","dataset":"Kinetics","model":"BIKE","rank_in_archive_order":8,"of":20,"metrics":{"Top-1 Accuracy":"68.5","Top-5 Accuracy":"91.1"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-action-recognition-on-ucf101","task":"Zero-Shot Action Recognition","dataset":"UCF101","model":"BIKE","rank_in_archive_order":5,"of":35,"metrics":{"Top-1 Accuracy":"86.6"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2301.00182","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}