{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/omnivl-one-foundation-model-for-image","title":"OmniVL:One Foundation Model for Image-Language and Video-Language Tasks","arxiv_id":"2209.07526","date":"2022-09-15","proceeding":null,"authors":["Junke Wang","Dongdong Chen","Zuxuan Wu","Chong Luo","Luowei Zhou","Yucheng Zhao","Yujia Xie","Ce Liu","Yu-Gang Jiang","Lu Yuan"],"abstract":"This paper presents OmniVL, a new foundation model to support both image-language and video-language tasks using one universal architecture. It adopts a unified transformer-based visual encoder for both image and video inputs, and thus can perform joint image-language and video-language pretraining. We demonstrate, for the first time, such a paradigm benefits both image and video tasks, as opposed to the conventional one-directional transfer (e.g., use image-language to help video-language). To this end, we propose a decoupled joint pretraining of image-language and video-language to effectively decompose the vision-language modeling into spatial and temporal dimensions and obtain performance boost on both image and video tasks. Moreover, we introduce a novel unified vision-language contrastive (UniVLC) loss to leverage image-text, video-text, image-label (e.g., image classification), video-label (e.g., video action recognition) data together, so that both supervised and noisily supervised pretraining data are utilized as much as possible. Without incurring extra task-specific adaptors, OmniVL can simultaneously support visual only tasks (e.g., image classification, video action recognition), cross-modal alignment tasks (e.g., image/video-text retrieval), and multi-modal understanding and generation tasks (e.g., image/video question answering, captioning). We evaluate OmniVL on a wide range of downstream tasks and achieve state-of-the-art or competitive results with similar model size and data scale.","url_abs":"https://arxiv.org/abs/2209.07526v2","url_pdf":"https://arxiv.org/pdf/2209.07526v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"action-classification","task_name":"Action Classification"},{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"cross-modal-retrieval","task_name":"Cross-Modal Retrieval"},{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"action-recognition","task_name":"Temporal Action Localization"},{"task_slug":"text-retrieval","task_name":"Text Retrieval"},{"task_slug":"video-captioning","task_name":"Video Captioning"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"},{"task_slug":"video-text-retrieval","task_name":"Video-Text Retrieval"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"zero-shot-video-retrieval","task_name":"Zero-Shot Video Retrieval"},{"task_slug":"cross-modal-alignment","task_name":"cross-modal alignment"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"OmniVL","rank_in_archive_order":117,"of":207,"metrics":{"Acc@1":"79.1","Acc@5":"94.5"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-something","task":"Action Recognition","dataset":"Something-Something V2","model":"OmniVL","rank_in_archive_order":102,"of":123,"metrics":{"Top-1 Accuracy":"62.5","Top-5 Accuracy":"86.2"},"uses_additional_data":false},{"leaderboard":"/sota/cross-modal-retrieval-on-coco-2014","task":"Cross-Modal Retrieval","dataset":"COCO 2014","model":"OmniVL (14M)","rank_in_archive_order":7,"of":36,"metrics":{"Image-to-text R@1":"82.1","Image-to-text R@10":"98.1","Image-to-text R@5":"95.9","Text-to-image R@1":"64.8","Text-to-image R@10":"91.6","Text-to-image R@5":"86.1"},"uses_additional_data":true},{"leaderboard":"/sota/cross-modal-retrieval-on-flickr30k","task":"Cross-Modal Retrieval","dataset":"Flickr30k","model":"OmniVL (14M)","rank_in_archive_order":4,"of":27,"metrics":{"Image-to-text R@1":"97.3","Image-to-text R@10":"100","Image-to-text R@5":"99.9","Text-to-image R@1":"87.9","Text-to-image R@10":"99.1","Text-to-image R@5":"97.8"},"uses_additional_data":true},{"leaderboard":"/sota/image-captioning-on-nocaps-val-in-domain","task":"Image Captioning","dataset":"nocaps-val-in-domain","model":"OmniVL","rank_in_archive_order":9,"of":11,"metrics":{"CIDEr":"104.6","Pre-train (#images)":"14M","SPICE":"15"},"uses_additional_data":false},{"leaderboard":"/sota/image-captioning-on-nocaps-val-near-domain","task":"Image Captioning","dataset":"nocaps-val-near-domain","model":"OmniVL","rank_in_archive_order":8,"of":10,"metrics":{"CIDEr":"108.3","Pre-train (#images)":"14M","SPICE":"14.9"},"uses_additional_data":false},{"leaderboard":"/sota/image-captioning-on-nocaps-val-out-domain","task":"Image Captioning","dataset":"nocaps-val-out-domain","model":"OmniVL","rank_in_archive_order":8,"of":10,"metrics":{"CIDEr":"106.3","Pretrain (#images)":"14M","SPICE":"14.2"},"uses_additional_data":false},{"leaderboard":"/sota/image-captioning-on-nocaps-val-overall","task":"Image Captioning","dataset":"nocaps-val-overall","model":"OmniVL","rank_in_archive_order":8,"of":11,"metrics":{"CIDEr":"107.5","Pretrain (#images)":"14M","SPICE":"14.7"},"uses_additional_data":false},{"leaderboard":"/sota/video-captioning-on-youcook2","task":"Video Captioning","dataset":"YouCook2","model":"OmniVL","rank_in_archive_order":11,"of":14,"metrics":{"BLEU-3":"12.87","BLEU-4":"8.72","CIDEr":"1.16","METEOR":"14.83","ROUGE-L":"36.09"},"uses_additional_data":false},{"leaderboard":"/sota/video-retrieval-on-didemo","task":"Video Retrieval","dataset":"DiDeMo","model":"OmniVL","rank_in_archive_order":21,"of":40,"metrics":{"text-to-video R@1":"52.4","text-to-video R@10":"85.4","text-to-video R@5":"79.5"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-msr-vtt","task":"Video Retrieval","dataset":"MSR-VTT","model":"OmniVL","rank_in_archive_order":13,"of":40,"metrics":{"text-to-video R@1":"47.8","text-to-video R@10":"83.8","text-to-video R@5":"74.2"},"uses_additional_data":true},{"leaderboard":"/sota/visual-question-answering-on-msrvtt-qa-1","task":"Visual Question Answering (VQA)","dataset":"MSRVTT-QA","model":"OmniVL","rank_in_archive_order":18,"of":34,"metrics":{"Accuracy":"0.441"},"uses_additional_data":true},{"leaderboard":"/sota/visual-question-answering-on-msvd-qa-1","task":"Visual Question Answering (VQA)","dataset":"MSVD-QA","model":"OmniVL","rank_in_archive_order":21,"of":36,"metrics":{"Accuracy":"0.510"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-didemo","task":"Zero-Shot Video Retrieval","dataset":"DiDeMo","model":"OmniVL","rank_in_archive_order":15,"of":26,"metrics":{"text-to-video R@1":"33.3","text-to-video R@10":"68.5","text-to-video R@5":"58.7"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-msr-vtt","task":"Zero-Shot Video Retrieval","dataset":"MSR-VTT","model":"OmniVL","rank_in_archive_order":18,"of":41,"metrics":{"text-to-video R@1":"34.6","text-to-video R@10":"66.6","text-to-video R@5":"58.4"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2209.07526","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}