{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ost-refining-text-knowledge-with-optimal","title":"OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition","arxiv_id":"2312.00096","date":"2023-11-30","proceeding":"CVPR 2024 1","authors":["Tongjia Chen","Hongshan Yu","Zhengeng Yang","Zechuan Li","Wei Sun","Chen Chen"],"abstract":"Due to the resource-intensive nature of training vision-language models on expansive video data, a majority of studies have centered on adapting pre-trained image-language models to the video domain. Dominant pipelines propose to tackle the visual discrepancies with additional temporal learners while overlooking the substantial discrepancy for web-scaled descriptive narratives and concise action category names, leading to less distinct semantic space and potential performance limitations. In this work, we prioritize the refinement of text knowledge to facilitate generalizable video recognition. To address the limitations of the less distinct semantic space of category names, we prompt a large language model (LLM) to augment action class names into Spatio-Temporal Descriptors thus bridging the textual discrepancy and serving as a knowledge base for general recognition. Moreover, to assign the best descriptors with different video instances, we propose Optimal Descriptor Solver, forming the video recognition problem as solving the optimal matching flow across frame-level representations and descriptors. Comprehensive evaluations in zero-shot, few-shot, and fully supervised video recognition highlight the effectiveness of our approach. Our best model achieves a state-of-the-art zero-shot accuracy of 75.1% on Kinetics-600.","url_abs":"https://arxiv.org/abs/2312.00096v2","url_pdf":"https://arxiv.org/pdf/2312.00096v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ost-refining-text-knowledge-with-optimal","repo_url":"https://github.com/tomchen-ctj/OST","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"descriptive","task_name":"Descriptive"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"video-recognition","task_name":"Video Recognition"},{"task_slug":"zero-shot-action-recognition","task_name":"Zero-Shot Action Recognition"},{"task_slug":null,"task_name":"Zero-Shot Action Recognition on HMDB51"},{"task_slug":null,"task_name":"Zero-Shot Action Recognition on UCF101"}],"methods":[{"method_slug":"base","method_name":"BASE"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/zero-shot-action-recognition-on-hmdb51","task":"Zero-Shot Action Recognition","dataset":"HMDB51","model":"OST","rank_in_archive_order":9,"of":29,"metrics":{"Top-1 Accuracy":"55.9"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-action-recognition-on-kinetics","task":"Zero-Shot Action Recognition","dataset":"Kinetics","model":"OST","rank_in_archive_order":3,"of":20,"metrics":{"Top-1 Accuracy":"75.1","Top-5 Accuracy":"94.6"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-action-recognition-on-ucf101","task":"Zero-Shot Action Recognition","dataset":"UCF101","model":"OST","rank_in_archive_order":10,"of":35,"metrics":{"Top-1 Accuracy":"79.7"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2312.00096","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.00096"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tomchen-ctj/OST","reach":null}],"summary":{"ran_fixture":1,"unverified":1},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b030778fdb39ccb1","entry":"gen_label","repo":"tomchen-ctj/OST","repo_kind":"official","path":"main_nce.py","file_url":"https://github.com/tomchen-ctj/OST/blob/HEAD/main_nce.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b030778fdb39ccb1"}},{"code_sha256_prefix":"08631eddcf79e1cc","entry":"gather_labels","repo":"tomchen-ctj/OST","repo_kind":"official","path":"main_nce.py","file_url":"https://github.com/tomchen-ctj/OST/blob/HEAD/main_nce.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"08631eddcf79e1cc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}