{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mmpt-multimodal-prompt-tuning-for-zero-shot","title":"M$^2$PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning","arxiv_id":"2409.15657","date":"2024-09-24","proceeding":null,"authors":["Taowen Wang","Yiyang Liu","James Chenhao Liang","Junhan Zhao","Yiming Cui","Yuning Mao","Shaoliang Nie","Jiahao Liu","Fuli Feng","Zenglin Xu","Cheng Han","Lifu Huang","Qifan Wang","Dongfang Liu"],"abstract":"Multimodal Large Language Models (MLLMs) demonstrate remarkable performance across a wide range of domains, with increasing emphasis on enhancing their zero-shot generalization capabilities for unseen tasks across various modalities. Instruction tuning has emerged as an effective strategy for achieving zero-shot generalization by finetuning pretrained models on diverse multimodal tasks. As the scale of MLLMs continues to grow, parameter-efficient finetuning becomes increasingly critical. However, most existing parameter-efficient approaches focus only on single modalities and often overlook the multimodal characteristics during finetuning. In this work, we introduce a novel Multimodal Prompt Tuning (M$^2$PT) approach for efficient instruction tuning of MLLMs. M$^2$PT effectively integrates visual and textual prompts into the vision encoder and language processor respectively during finetuning, facilitating the extraction and alignment of features across modalities. Empirical results on various multimodal evaluation datasets demonstrate the superior performance of our approach compared to several state-of-the-art baselines. A comprehensive set of ablation studies validates the effectiveness of our prompt design and the efficiency of our approach.","url_abs":"https://arxiv.org/abs/2409.15657v4","url_pdf":"https://arxiv.org/pdf/2409.15657v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mmpt-multimodal-prompt-tuning-for-zero-shot","repo_url":"https://github.com/william-wang618/m2pt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"mmpt-multimodal-prompt-tuning-for-zero-shot","repo_url":"https://github.com/william-wang618/mmpt-emnlp2024","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"zero-shot-generalization","task_name":"Zero-shot Generalization"}],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2409.15657","atlas_url":"https://app.syntology.ai/?focus=2409.15657","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.15657"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/william-wang618/mmpt-emnlp2024","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/william-wang618/m2pt","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models","reach":{"status":"ok"}}],"summary":{"ran":10,"ran_draft_wrong":6,"unverified":10},"by_repo_kind":{"official":{"samples":26,"ran":16,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":26,"samples":[{"code_sha256_prefix":"2fac88833404a422","entry":"CLIPAttention","repo":"william-wang618/m2pt","repo_kind":"official","path":"M2PT/model/llava_archPT.py","file_url":"https://github.com/william-wang618/m2pt/blob/HEAD/M2PT/model/llava_archPT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2fac88833404a422"}},{"code_sha256_prefix":"4ea5ab8d39e37583","entry":"CLIPMLP","repo":"william-wang618/m2pt","repo_kind":"official","path":"M2PT/model/llava_archPT.py","file_url":"https://github.com/william-wang618/m2pt/blob/HEAD/M2PT/model/llava_archPT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4ea5ab8d39e37583"}},{"code_sha256_prefix":"295ac6633a3d6e3f","entry":"CLIPOutput","repo":"william-wang618/m2pt","repo_kind":"official","path":"M2PT/model/llava_archPT.py","file_url":"https://github.com/william-wang618/m2pt/blob/HEAD/M2PT/model/llava_archPT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"295ac6633a3d6e3f"}},{"code_sha256_prefix":"2200675ee7e0f4c5","entry":"CLIPTextEmbeddings","repo":"william-wang618/m2pt","repo_kind":"official","path":"M2PT/model/llava_archPT.py","file_url":"https://github.com/william-wang618/m2pt/blob/HEAD/M2PT/model/llava_archPT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2200675ee7e0f4c5"}},{"code_sha256_prefix":"1aa764d481521c7c","entry":"CLIPTextModelOutput","repo":"william-wang618/m2pt","repo_kind":"official","path":"M2PT/model/llava_archPT.py","file_url":"https://github.com/william-wang618/m2pt/blob/HEAD/M2PT/model/llava_archPT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1aa764d481521c7c"}},{"code_sha256_prefix":"17e8f9a17c97b609","entry":"CLIPVisionEmbeddings","repo":"william-wang618/m2pt","repo_kind":"official","path":"M2PT/model/llava_archPT.py","file_url":"https://github.com/william-wang618/m2pt/blob/HEAD/M2PT/model/llava_archPT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"17e8f9a17c97b609"}},{"code_sha256_prefix":"7a4d14a0c2a66b91","entry":"CLIPVisionModel","repo":"william-wang618/m2pt","repo_kind":"official","path":"M2PT/model/llava_archPT.py","file_url":"https://github.com/william-wang618/m2pt/blob/HEAD/M2PT/model/llava_archPT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7a4d14a0c2a66b91"}},{"code_sha256_prefix":"bae888fcd5c5473f","entry":"CLIPVisionModelOutput","repo":"william-wang618/m2pt","repo_kind":"official","path":"M2PT/model/llava_archPT.py","file_url":"https://github.com/william-wang618/m2pt/blob/HEAD/M2PT/model/llava_archPT.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bae888fcd5c5473f"}},{"code_sha256_prefix":"0bac4ffb06162f12","entry":"_expand_mask","repo":"william-wang618/m2pt","repo_kind":"official","path":"M2PT/model/llava_archPT.py","file_url":"https://github.com/william-wang618/m2pt/blob/HEAD/M2PT/model/llava_archPT.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0bac4ffb06162f12"}},{"code_sha256_prefix":"017dd63ed48e15f1","entry":"clip_loss","repo":"william-wang618/m2pt","repo_kind":"official","path":"M2PT/model/llava_archPT.py","file_url":"https://github.com/william-wang618/m2pt/blob/HEAD/M2PT/model/llava_archPT.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"017dd63ed48e15f1"}},{"code_sha256_prefix":"20e4f665698a3d18","entry":"collate_fn","repo":"william-wang618/mmpt-emnlp2024","repo_kind":"official","path":"M2PT/eval/model_vqa_loader_PT_mme.py","file_url":"https://github.com/william-wang618/mmpt-emnlp2024/blob/HEAD/M2PT/eval/model_vqa_loader_PT_mme.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"20e4f665698a3d18"}},{"code_sha256_prefix":"35caa662e2a8cf43","entry":"contrastive_loss","repo":"william-wang618/m2pt","repo_kind":"official","path":"M2PT/model/llava_archPT.py","file_url":"https://github.com/william-wang618/m2pt/blob/HEAD/M2PT/model/llava_archPT.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"35caa662e2a8cf43"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"william-wang618/mmpt-emnlp2024","repo_kind":"official","path":"M2PT/eval/model_vqa.py","file_url":"https://github.com/william-wang618/mmpt-emnlp2024/blob/HEAD/M2PT/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"c3ee9d07c900dd55","entry":"load_image_from_base64","repo":"william-wang618/mmpt-emnlp2024","repo_kind":"official","path":"M2PT/mm_utils.py","file_url":"https://github.com/william-wang618/mmpt-emnlp2024/blob/HEAD/M2PT/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c3ee9d07c900dd55"}},{"code_sha256_prefix":"1df990c375318896","entry":"process_images","repo":"william-wang618/mmpt-emnlp2024","repo_kind":"official","path":"M2PT/mm_utils.py","file_url":"https://github.com/william-wang618/mmpt-emnlp2024/blob/HEAD/M2PT/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1df990c375318896"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"william-wang618/mmpt-emnlp2024","repo_kind":"official","path":"M2PT/eval/model_vqa.py","file_url":"https://github.com/william-wang618/mmpt-emnlp2024/blob/HEAD/M2PT/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"934679f6a65cc3f3","entry":"CLIPEncoder","repo":"william-wang618/m2pt","repo_kind":"official","path":"M2PT/model/llava_archPT.py","file_url":"https://github.com/william-wang618/m2pt/blob/HEAD/M2PT/model/llava_archPT.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"934679f6a65cc3f3"}},{"code_sha256_prefix":"5e590d2924823b50","entry":"CLIPEncoderLayer","repo":"william-wang618/m2pt","repo_kind":"official","path":"M2PT/model/llava_archPT.py","file_url":"https://github.com/william-wang618/m2pt/blob/HEAD/M2PT/model/llava_archPT.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5e590d2924823b50"}},{"code_sha256_prefix":"88581d28cc6e056a","entry":"CLIPModel","repo":"william-wang618/m2pt","repo_kind":"official","path":"M2PT/model/llava_archPT.py","file_url":"https://github.com/william-wang618/m2pt/blob/HEAD/M2PT/model/llava_archPT.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"88581d28cc6e056a"}},{"code_sha256_prefix":"6299098931d7cfdb","entry":"CLIPTextModelWithProjection","repo":"william-wang618/m2pt","repo_kind":"official","path":"M2PT/model/llava_archPT.py","file_url":"https://github.com/william-wang618/m2pt/blob/HEAD/M2PT/model/llava_archPT.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6299098931d7cfdb"}},{"code_sha256_prefix":"0ee6d16d12c11f15","entry":"CLIPTextTransformer","repo":"william-wang618/m2pt","repo_kind":"official","path":"M2PT/model/llava_archPT.py","file_url":"https://github.com/william-wang618/m2pt/blob/HEAD/M2PT/model/llava_archPT.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0ee6d16d12c11f15"}},{"code_sha256_prefix":"129981f351dd0e2c","entry":"CLIPVisionModelWithProjection","repo":"william-wang618/m2pt","repo_kind":"official","path":"M2PT/model/llava_archPT.py","file_url":"https://github.com/william-wang618/m2pt/blob/HEAD/M2PT/model/llava_archPT.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"129981f351dd0e2c"}},{"code_sha256_prefix":"f08131025c9de6a1","entry":"CLIPVisionTower","repo":"william-wang618/m2pt","repo_kind":"official","path":"M2PT/model/llava_archPT.py","file_url":"https://github.com/william-wang618/m2pt/blob/HEAD/M2PT/model/llava_archPT.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f08131025c9de6a1"}},{"code_sha256_prefix":"d942bd5ff461ea13","entry":"CLIPVisionTransformer","repo":"william-wang618/m2pt","repo_kind":"official","path":"M2PT/model/llava_archPT.py","file_url":"https://github.com/william-wang618/m2pt/blob/HEAD/M2PT/model/llava_archPT.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d942bd5ff461ea13"}},{"code_sha256_prefix":"959e688412f8624f","entry":"LlavaMetaModel","repo":"william-wang618/m2pt","repo_kind":"official","path":"M2PT/model/llava_archPT.py","file_url":"https://github.com/william-wang618/m2pt/blob/HEAD/M2PT/model/llava_archPT.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"959e688412f8624f"}},{"code_sha256_prefix":"a11dbce7f6abb48d","entry":"build_vision_tower","repo":"william-wang618/m2pt","repo_kind":"official","path":"M2PT/model/llava_archPT.py","file_url":"https://github.com/william-wang618/m2pt/blob/HEAD/M2PT/model/llava_archPT.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a11dbce7f6abb48d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}