{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vlmpc-vision-language-model-predictive","title":"VLMPC: Vision-Language Model Predictive Control for Robotic Manipulation","arxiv_id":"2407.09829","date":"2024-07-13","proceeding":null,"authors":["Wentao Zhao","Jiaming Chen","Ziyu Meng","Donghui Mao","Ran Song","Wei Zhang"],"abstract":"Although Model Predictive Control (MPC) can effectively predict the future states of a system and thus is widely used in robotic manipulation tasks, it does not have the capability of environmental perception, leading to the failure in some complex scenarios. To address this issue, we introduce Vision-Language Model Predictive Control (VLMPC), a robotic manipulation framework which takes advantage of the powerful perception capability of vision language model (VLM) and integrates it with MPC. Specifically, we propose a conditional action sampling module which takes as input a goal image or a language instruction and leverages VLM to sample a set of candidate action sequences. Then, a lightweight action-conditioned video prediction model is designed to generate a set of future frames conditioned on the candidate action sequences. VLMPC produces the optimal action sequence with the assistance of VLM through a hierarchical cost function that formulates both pixel-level and knowledge-level consistence between the current observation and the goal image. We demonstrate that VLMPC outperforms the state-of-the-art methods on public benchmarks. More importantly, our method showcases excellent performance in various real-world tasks of robotic manipulation. Code is available at~\\url{https://github.com/PPjmchen/VLMPC}.","url_abs":"https://arxiv.org/abs/2407.09829v1","url_pdf":"https://arxiv.org/pdf/2407.09829v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"vlmpc-vision-language-model-predictive","repo_url":"https://github.com/ppjmchen/vlmpc","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.09829","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.09829"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ppjmchen/vlmpc","reach":{"status":"ok"}}],"summary":{"ran":13,"unverified":2},"by_repo_kind":{"official":{"samples":15,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":15,"samples":[{"code_sha256_prefix":"f10987ef9f8a3ec9","entry":"bbox_convert_vert_to_xywh","repo":"ppjmchen/vlmpc","repo_kind":"official","path":"Traj_vlmpc/tools.py","file_url":"https://github.com/ppjmchen/vlmpc/blob/HEAD/Traj_vlmpc/tools.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f10987ef9f8a3ec9"}},{"code_sha256_prefix":"aa1c1367c26003ae","entry":"calculate_iou","repo":"ppjmchen/vlmpc","repo_kind":"official","path":"vlmpc.py","file_url":"https://github.com/ppjmchen/vlmpc/blob/HEAD/vlmpc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa1c1367c26003ae"}},{"code_sha256_prefix":"48f0287ceccd04b7","entry":"compute_dis","repo":"ppjmchen/vlmpc","repo_kind":"official","path":"vlmpc.py","file_url":"https://github.com/ppjmchen/vlmpc/blob/HEAD/vlmpc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"48f0287ceccd04b7"}},{"code_sha256_prefix":"fb5aeab301ed8571","entry":"convert_env_action","repo":"ppjmchen/vlmpc","repo_kind":"official","path":"vlmpc.py","file_url":"https://github.com/ppjmchen/vlmpc/blob/HEAD/vlmpc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fb5aeab301ed8571"}},{"code_sha256_prefix":"5a2b18595a5ea8b4","entry":"det_bbox","repo":"ppjmchen/vlmpc","repo_kind":"official","path":"detect_bbx.py","file_url":"https://github.com/ppjmchen/vlmpc/blob/HEAD/detect_bbx.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5a2b18595a5ea8b4"}},{"code_sha256_prefix":"7d3fdb8d8156ad15","entry":"dict_to_numpy","repo":"ppjmchen/vlmpc","repo_kind":"official","path":"Traj_vlmpc/tools.py","file_url":"https://github.com/ppjmchen/vlmpc/blob/HEAD/Traj_vlmpc/tools.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7d3fdb8d8156ad15"}},{"code_sha256_prefix":"a6dcc893935ac198","entry":"downsample_point_cloud","repo":"ppjmchen/vlmpc","repo_kind":"official","path":"Traj_vlmpc/image2robot.py","file_url":"https://github.com/ppjmchen/vlmpc/blob/HEAD/Traj_vlmpc/image2robot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a6dcc893935ac198"}},{"code_sha256_prefix":"2512f3efe01084d7","entry":"generate_point_cloud","repo":"ppjmchen/vlmpc","repo_kind":"official","path":"Traj_vlmpc/cost_function.py","file_url":"https://github.com/ppjmchen/vlmpc/blob/HEAD/Traj_vlmpc/cost_function.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2512f3efe01084d7"}},{"code_sha256_prefix":"6ab562f2ab84a1cb","entry":"get_interactive_object","repo":"ppjmchen/vlmpc","repo_kind":"official","path":"prompt_gpt.py","file_url":"https://github.com/ppjmchen/vlmpc/blob/HEAD/prompt_gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6ab562f2ab84a1cb"}},{"code_sha256_prefix":"27c6c35d51904ddc","entry":"image2robot_mech","repo":"ppjmchen/vlmpc","repo_kind":"official","path":"Traj_vlmpc/image2robot.py","file_url":"https://github.com/ppjmchen/vlmpc/blob/HEAD/Traj_vlmpc/image2robot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"27c6c35d51904ddc"}},{"code_sha256_prefix":"6368d1ac894a698d","entry":"redistribute_points","repo":"ppjmchen/vlmpc","repo_kind":"official","path":"Traj_vlmpc/mixed_gaussian.py","file_url":"https://github.com/ppjmchen/vlmpc/blob/HEAD/Traj_vlmpc/mixed_gaussian.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6368d1ac894a698d"}},{"code_sha256_prefix":"36da76419a022fff","entry":"setup_logger","repo":"ppjmchen/vlmpc","repo_kind":"official","path":"logger.py","file_url":"https://github.com/ppjmchen/vlmpc/blob/HEAD/logger.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"36da76419a022fff"}},{"code_sha256_prefix":"21fa9e59351465b5","entry":"tf_collate_fn","repo":"ppjmchen/vlmpc","repo_kind":"official","path":"Traj_vlmpc/tools.py","file_url":"https://github.com/ppjmchen/vlmpc/blob/HEAD/Traj_vlmpc/tools.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"21fa9e59351465b5"}},{"code_sha256_prefix":"7f1e466c08653b61","entry":"get_subtask_make_line","repo":"ppjmchen/vlmpc","repo_kind":"official","path":"prompt_gpt.py","file_url":"https://github.com/ppjmchen/vlmpc/blob/HEAD/prompt_gpt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7f1e466c08653b61"}},{"code_sha256_prefix":"da82ddf559187f61","entry":"get_subtask_push_corner","repo":"ppjmchen/vlmpc","repo_kind":"official","path":"prompt_gpt.py","file_url":"https://github.com/ppjmchen/vlmpc/blob/HEAD/prompt_gpt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"da82ddf559187f61"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}