{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/enabling-multimodal-generation-on-clip-via","title":"Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation","arxiv_id":null,"date":"2021-11-16","proceeding":"ACL ARR November 2021 11","authors":["Anonymous"],"abstract":"The recent large-scale vision-language pre-training (VLP) of dual-stream architectures (e.g., CLIP) with a tremendous amount of image-text pair data, has shown its superiority on various multimodal alignment tasks. Despite its success, the resulting models are not capable of generative multimodal tasks due to the weak text encoder. To tackle this problem, we propose to augment the dual-stream VLP model with a textual pre-trained language model (PLM) via vision-language knowledge distillation (VLKD), enabling the capability for multimodal generation. VLKD is pretty data- and computation-efficient compared to the pre-training from scratch. Experimental results show that the resulting model has strong zero-shot performance on multimodal generation tasks, such as open-ended visual question answering and image captioning. For example, it achieves 39.7% zero-shot accuracy on the VQA 2.0 dataset, surpassing the previous state-of-the-art zero-shot model with 14x fewer parameters. Furthermore, the original text processing ability of the PLM is maintained after VLKD, which makes our model versatile for both multimodal and unimodal tasks.","url_abs":"https://openreview.net/forum?id=YTGg7kv8qIq","url_pdf":"https://openreview.net/pdf?id=YTGg7kv8qIq","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"multimodal-generation","task_name":"multimodal generation"}],"methods":[{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-captioning-on-coco-captions","task":"Image Captioning","dataset":"COCO Captions","model":"VLKD (ViT-B/16)","rank_in_archive_order":34,"of":41,"metrics":{"BLEU-4":"16.7","CIDER":"58.3","METEOR":"19.7","SPICE":"13.4"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-ok-vqa","task":"Visual Question Answering (VQA)","dataset":"OK-VQA","model":"VLKD(ViT-B/16)","rank_in_archive_order":36,"of":37,"metrics":{"Accuracy":"10.5"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-vqa-v2-test-dev","task":"Visual Question Answering (VQA)","dataset":"VQA v2 test-dev","model":"VLKD","rank_in_archive_order":56,"of":56,"metrics":{"Accuracy":"44.5"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-vqa-v2-val","task":"Visual Question Answering (VQA)","dataset":"VQA v2 val","model":"VLKD(ViT-B/16)","rank_in_archive_order":10,"of":11,"metrics":{"Accuracy":"38.6"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}