{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/prompt-tuning-for-generative-multimodal","title":"Prompt Tuning for Generative Multimodal Pretrained Models","arxiv_id":"2208.02532","date":"2022-08-04","proceeding":null,"authors":["Hao Yang","Junyang Lin","An Yang","Peng Wang","Chang Zhou","Hongxia Yang"],"abstract":"Prompt tuning has become a new paradigm for model tuning and it has demonstrated success in natural language pretraining and even vision pretraining. In this work, we explore the transfer of prompt tuning to multimodal pretraining, with a focus on generative multimodal pretrained models, instead of contrastive ones. Specifically, we implement prompt tuning on the unified sequence-to-sequence pretrained model adaptive to both understanding and generation tasks. Experimental results demonstrate that the light-weight prompt tuning can achieve comparable performance with finetuning and surpass other light-weight tuning methods. Besides, in comparison with finetuned models, the prompt-tuned models demonstrate improved robustness against adversarial attacks. We further figure out that experimental factors, including the prompt length, prompt depth, and reparameteratization, have great impacts on the model performance, and thus we empirically provide a recommendation for the setups of prompt tuning. Despite the observed advantages, we still find some limitations in prompt tuning, and we correspondingly point out the directions for future studies. Codes are available at \\url{https://github.com/OFA-Sys/OFA}","url_abs":"https://arxiv.org/abs/2208.02532v1","url_pdf":"https://arxiv.org/pdf/2208.02532v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"prompt-tuning-for-generative-multimodal","repo_url":"https://github.com/ofa-sys/ofa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-captioning","task_name":"Image Captioning"},{"task_slug":"visual-entailment","task_name":"Visual Entailment"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-captioning-on-coco-captions","task":"Image Captioning","dataset":"COCO Captions","model":"Prompt Tuning","rank_in_archive_order":10,"of":41,"metrics":{"BLEU-4":"41.81","CIDER":"141.4","METEOR":"31.51","SPICE":"24.42"},"uses_additional_data":false},{"leaderboard":"/sota/visual-entailment-on-snli-ve-test","task":"Visual Entailment","dataset":"SNLI-VE test","model":"Prompt Tuning","rank_in_archive_order":2,"of":8,"metrics":{"Accuracy":"90.12"},"uses_additional_data":false},{"leaderboard":"/sota/visual-entailment-on-snli-ve-val","task":"Visual Entailment","dataset":"SNLI-VE val","model":"Prompt Tuning","rank_in_archive_order":2,"of":9,"metrics":{"Accuracy":"90.04"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-vqa-v2-test-std","task":"Visual Question Answering (VQA)","dataset":"VQA v2 test-std","model":"Prompt Tuning","rank_in_archive_order":10,"of":38,"metrics":{"overall":"78.53"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2208.02532","atlas_url":"https://app.syntology.ai/?focus=2208.02532","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2208.02532"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ofa-sys/ofa","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_honours":1},"by_repo_kind":{},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"357d3a4f0be679d9","entry":"apply_half","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"357d3a4f0be679d9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}