{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hitea-hierarchical-temporal-aware-video","title":"HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training","arxiv_id":"2212.14546","date":"2022-12-30","proceeding":"ICCV 2023 1","authors":["Qinghao Ye","Guohai Xu","Ming Yan","Haiyang Xu","Qi Qian","Ji Zhang","Fei Huang"],"abstract":"Video-language pre-training has advanced the performance of various downstream video-language tasks. However, most previous methods directly inherit or adapt typical image-language pre-training paradigms to video-language pre-training, thus not fully exploiting the unique characteristic of video, i.e., temporal. In this paper, we propose a Hierarchical Temporal-Aware video-language pre-training framework, HiTeA, with two novel pre-training tasks for modeling cross-modal alignment between moments and texts as well as the temporal relations of video-text pairs. Specifically, we propose a cross-modal moment exploration task to explore moments in videos, which results in detailed video moment representation. Besides, the inherent temporal relations are captured by aligning video-text pairs as a whole in different time resolutions with multi-modal temporal relation exploration task. Furthermore, we introduce the shuffling test to evaluate the temporal reliance of datasets and video-language pre-training models. We achieve state-of-the-art results on 15 well-established video-language understanding and generation tasks, especially on temporal-oriented datasets (e.g., SSv2-Template and SSv2-Label) with 8.6% and 11.1% improvement respectively. HiTeA also demonstrates strong generalization ability when directly transferred to downstream tasks in a zero-shot manner. Models and demo will be available on ModelScope.","url_abs":"https://arxiv.org/abs/2212.14546v1","url_pdf":"https://arxiv.org/pdf/2212.14546v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"tgif-action","task_name":"TGIF-Action"},{"task_slug":"tgif-frame","task_name":"TGIF-Frame"},{"task_slug":"tgif-transition","task_name":"TGIF-Transition"},{"task_slug":"video-captioning","task_name":"Video Captioning"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"},{"task_slug":"zero-shot-video-retrieval","task_name":"Zero-Shot Video Retrieval"},{"task_slug":"cross-modal-alignment","task_name":"cross-modal alignment"}],"methods":[{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-captioning-on-msr-vtt-1","task":"Video Captioning","dataset":"MSR-VTT","model":"HiTeA","rank_in_archive_order":11,"of":24,"metrics":{"BLEU-4":"49.2","CIDEr":"65.1","METEOR":"30.7","ROUGE-L":"65.0"},"uses_additional_data":true},{"leaderboard":"/sota/video-captioning-on-msvd-1","task":"Video Captioning","dataset":"MSVD","model":"HiTeA","rank_in_archive_order":7,"of":16,"metrics":{"BLEU-4":"71.0","CIDEr":"146.9","METEOR":"45.3","ROUGE-L":"81.4"},"uses_additional_data":true},{"leaderboard":"/sota/video-question-answering-on-msrvtt-mc","task":"Video Question Answering","dataset":"MSRVTT-MC","model":"HiTeA","rank_in_archive_order":2,"of":7,"metrics":{"Accuracy":"97.4"},"uses_additional_data":false},{"leaderboard":"/sota/video-question-answering-on-next-qa","task":"Video Question Answering","dataset":"NExT-QA","model":"HiTeA","rank_in_archive_order":33,"of":47,"metrics":{"Accuracy":"63.1"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-activitynet","task":"Video Retrieval","dataset":"ActivityNet","model":"HiTeA","rank_in_archive_order":17,"of":31,"metrics":{"text-to-video R@1":"49.7","text-to-video R@10":"86.7","text-to-video R@5":"77.1"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-didemo","task":"Video Retrieval","dataset":"DiDeMo","model":"HiTeA","rank_in_archive_order":13,"of":40,"metrics":{"text-to-video R@1":"56.5","text-to-video R@10":"89.7","text-to-video R@5":"81.7"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-lsmdc","task":"Video Retrieval","dataset":"LSMDC","model":"HiTeA","rank_in_archive_order":12,"of":38,"metrics":{"text-to-video R@1":"28.7","text-to-video R@10":"59.0","text-to-video R@5":"50.3"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-msr-vtt-1ka","task":"Video Retrieval","dataset":"MSR-VTT-1kA","model":"HiTeA","rank_in_archive_order":33,"of":63,"metrics":{"text-to-video R@1":"46.8","text-to-video R@10":"81.9","text-to-video R@5":"71.2"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-ssv2-label-retrieval","task":"Video Retrieval","dataset":"SSv2-label retrieval","model":"HiTeA","rank_in_archive_order":3,"of":5,"metrics":{"text-to-video R@1":"55.2","text-to-video R@10":"81.4","text-to-video R@5":"89.1"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-ssv2-template-retrieval","task":"Video Retrieval","dataset":"SSv2-template retrieval","model":"HiTeA","rank_in_archive_order":3,"of":5,"metrics":{"text-to-video R@1":"85.6","text-to-video R@10":"100","text-to-video R@5":"100"},"uses_additional_data":true},{"leaderboard":"/sota/visual-question-answering-on-msrvtt-qa-1","task":"Visual Question Answering (VQA)","dataset":"MSRVTT-QA","model":"HiTeA","rank_in_archive_order":12,"of":34,"metrics":{"Accuracy":"0.459"},"uses_additional_data":true},{"leaderboard":"/sota/visual-question-answering-on-msvd-qa-1","task":"Visual Question Answering (VQA)","dataset":"MSVD-QA","model":"HiTeA","rank_in_archive_order":11,"of":36,"metrics":{"Accuracy":"0.556"},"uses_additional_data":true},{"leaderboard":"/sota/visual-question-answering-on-tgif-qa","task":"Visual Question Answering (VQA)","dataset":"TGIF-QA","model":"HiTeA","rank_in_archive_order":1,"of":2,"metrics":{"Accuracy":"0.732"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-learning-on-msrvtt-qa","task":"Zero-Shot Learning","dataset":"MSRVTT-QA","model":"HiTeA","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy":"21.7"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-learning-on-msvd-qa","task":"Zero-Shot Learning","dataset":"MSVD-QA","model":"HiTeA","rank_in_archive_order":1,"of":1,"metrics":{"Accuracy":"37.4"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-didemo","task":"Zero-Shot Video Retrieval","dataset":"DiDeMo","model":"HiTeA-17M","rank_in_archive_order":8,"of":26,"metrics":{"text-to-video R@1":"43.2","text-to-video R@10":"79.0","text-to-video R@5":"69.3"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-didemo","task":"Zero-Shot Video Retrieval","dataset":"DiDeMo","model":"HiTeA-5M","rank_in_archive_order":13,"of":26,"metrics":{"text-to-video R@1":"36.1","text-to-video R@10":"70.3","text-to-video R@5":"60.1"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-lsmdc","task":"Zero-Shot Video Retrieval","dataset":"LSMDC","model":"HiTeA-17M","rank_in_archive_order":7,"of":16,"metrics":{"text-to-video R@1":"18.3","text-to-video R@10":"44.2","text-to-video R@5":"36.7"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-lsmdc","task":"Zero-Shot Video Retrieval","dataset":"LSMDC","model":"HiTeA-5M","rank_in_archive_order":11,"of":16,"metrics":{"text-to-video R@1":"15.5","text-to-video R@10":"39.8","text-to-video R@5":"31.1"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-msr-vtt","task":"Zero-Shot Video Retrieval","dataset":"MSR-VTT","model":"HiTeA-17M","rank_in_archive_order":19,"of":41,"metrics":{"text-to-video R@1":"34.4","text-to-video R@10":"69.9","text-to-video R@5":"60.0"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-video-retrieval-on-msr-vtt","task":"Zero-Shot Video Retrieval","dataset":"MSR-VTT","model":"HiTeA-5M","rank_in_archive_order":23,"of":41,"metrics":{"text-to-video R@1":"29.9","text-to-video R@10":"62.9","text-to-video R@5":"54.2"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2212.14546","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}