{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2510-09201","title":"Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs","arxiv_id":"2510.09201","date":"2025-10-10","proceeding":"ICLR","authors":["Yumin Choi","Dongki Kim","Jinheon Baek","Sung Ju Hwang"],"abstract":"Large Language Models (LLMs) have shown remarkable success, and their multimodal expansions (MLLMs) further unlock capabilities spanning images, videos, and other modalities beyond text. However, despite this shift, prompt optimization approaches, designed to reduce the burden of manual prompt crafting while maximizing performance, remain confined to text, ultimately limiting the full potential of MLLMs. Motivated by this gap, we introduce the new problem of multimodal prompt optimization, which expands the prior definition of prompt optimization to the multimodal space defined by the pairs of textual and non-textual prompts. To tackle this problem, we then propose the Multimodal Prompt Optimizer (MPO), a unified framework that not only performs the joint optimization of multimodal prompts through alignment-preserving updates but also guides the selection process of candidate prompts by leveraging earlier evaluations as priors in a Bayesian-based selection strategy. Through extensive experiments across diverse modalities that go beyond text, such as images, videos, and even molecules, we demonstrate that MPO outperforms leading text-only optimization methods, establishing multimodal prompt optimization as a crucial step to realizing the potential of MLLMs.","url_abs":"https://arxiv.org/abs/2510.09201","url_pdf":"https://arxiv.org/pdf/2510.09201","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2510.09201","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2510.09201"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/Dozi01/MPO","reach":{"status":"ok"}}],"summary":{"ran_violates":1,"unverified":5},"by_repo_kind":{"found_in_text":{"samples":6,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"7c508037b40522af","entry":"str2bool","repo":"Dozi01/MPO","repo_kind":"found_in_text","path":"src/utils.py","file_url":"https://github.com/Dozi01/MPO/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7c508037b40522af"}},{"code_sha256_prefix":"348b87e4315c396d","entry":"check_mm_type","repo":"Dozi01/MPO","repo_kind":"found_in_text","path":"src/utils.py","file_url":"https://github.com/Dozi01/MPO/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"348b87e4315c396d"}},{"code_sha256_prefix":"aa1a526e833d1625","entry":"create_logger","repo":"Dozi01/MPO","repo_kind":"found_in_text","path":"src/utils.py","file_url":"https://github.com/Dozi01/MPO/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa1a526e833d1625"}},{"code_sha256_prefix":"d81cf93dfced14cf","entry":"get_multimodal_analysis_prompt","repo":"Dozi01/MPO","repo_kind":"found_in_text","path":"src/optim_model.py","file_url":"https://github.com/Dozi01/MPO/blob/HEAD/src/optim_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d81cf93dfced14cf"}},{"code_sha256_prefix":"10925e2ecaf7b13d","entry":"get_multimodal_edit_prompt","repo":"Dozi01/MPO","repo_kind":"found_in_text","path":"src/optim_model.py","file_url":"https://github.com/Dozi01/MPO/blob/HEAD/src/optim_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"10925e2ecaf7b13d"}},{"code_sha256_prefix":"9b170ed09f2d3739","entry":"get_multimodal_generation_prompt","repo":"Dozi01/MPO","repo_kind":"found_in_text","path":"src/optim_model.py","file_url":"https://github.com/Dozi01/MPO/blob/HEAD/src/optim_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b170ed09f2d3739"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_api"},"syntology_extracted_results":null}