{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/2505-11409","title":"Visual Planning: Let's Think Only with Images","arxiv_id":"2505.11409","date":"2025-05-16","proceeding":null,"authors":["Yi Xu","Chengzu Li","Han Zhou","Xingchen Wan","Caiqi Zhang","Anna Korhonen","Ivan Vulić"],"abstract":"Recent advancements in Large Language Models (LLMs) and their multimodal extensions (MLLMs) have substantially enhanced machine reasoning across diverse tasks. However, these models predominantly rely on pure text as the medium for both expressing and structuring reasoning, even when visual information is present. In this work, we argue that language may not always be the most natural or effective modality for reasoning, particularly in tasks involving spatial and geometrical information. Motivated by this, we propose a new paradigm, Visual Planning, which enables planning through purely visual representations, independent of text. In this paradigm, planning is executed via sequences of images that encode step-by-step inference in the visual domain, akin to how humans sketch or visualize future actions. We introduce a novel reinforcement learning framework, Visual Planning via Reinforcement Learning (VPRL), empowered by GRPO for post-training large vision models, leading to substantial improvements in planning in a selection of representative visual navigation tasks, FrozenLake, Maze, and MiniBehavior. Our visual planning paradigm outperforms all other planning variants that conduct reasoning in the text-only space. Our results establish Visual Planning as a viable and promising alternative to language-based reasoning, opening new avenues for tasks that benefit from intuitive, image-based inference.","url_abs":"https://arxiv.org/abs/2505.11409v1","url_pdf":"https://arxiv.org/pdf/2505.11409v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"2505-11409","repo_url":"https://github.com/yix8/visualplanning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"visual-navigation","task_name":"Visual Navigation"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2505.11409","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.11409"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/yix8/VisualPlanning","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yix8/visualplanning","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran_honours":2,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5e8624e67339cc75","entry":"action_reward_func","repo":"yix8/VisualPlanning","repo_kind":"official","path":"train_rl_frozen.py","file_url":"https://github.com/yix8/VisualPlanning/blob/HEAD/train_rl_frozen.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e8624e67339cc75"}},{"code_sha256_prefix":"85cea4702c2b11ca","entry":"action_reward_func","repo":"yix8/VisualPlanning","repo_kind":"official","path":"train_rl_mini.py","file_url":"https://github.com/yix8/VisualPlanning/blob/HEAD/train_rl_mini.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"85cea4702c2b11ca"}},{"code_sha256_prefix":"f55c31d782e48ffb","entry":"collate_fn","repo":"yix8/VisualPlanning","repo_kind":"official","path":"train_rl_frozen.py","file_url":"https://github.com/yix8/VisualPlanning/blob/HEAD/train_rl_frozen.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f55c31d782e48ffb"}},{"code_sha256_prefix":"d83bb7c4309631d8","entry":"int_reward_func","repo":"yix8/VisualPlanning","repo_kind":"official","path":"train_rl_frozen.py","file_url":"https://github.com/yix8/VisualPlanning/blob/HEAD/train_rl_frozen.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d83bb7c4309631d8"}},{"code_sha256_prefix":"dd3480b42b026abb","entry":"action_reward_func","repo":"yix8/VisualPlanning","repo_kind":"official","path":"train_rl_maze.py","file_url":"https://github.com/yix8/VisualPlanning/blob/HEAD/train_rl_maze.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dd3480b42b026abb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}