{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/planning-as-in-painting-a-diffusion-based","title":"Planning as In-Painting: A Diffusion-Based Embodied Task Planning Framework for Environments under Uncertainty","arxiv_id":"2312.01097","date":"2023-12-02","proceeding":null,"authors":["Cheng-Fu Yang","Haoyang Xu","Te-Lin Wu","Xiaofeng Gao","Kai-Wei Chang","Feng Gao"],"abstract":"Task planning for embodied AI has been one of the most challenging problems where the community does not meet a consensus in terms of formulation. In this paper, we aim to tackle this problem with a unified framework consisting of an end-to-end trainable method and a planning algorithm. Particularly, we propose a task-agnostic method named 'planning as in-painting'. In this method, we use a Denoising Diffusion Model (DDM) for plan generation, conditioned on both language instructions and perceptual inputs under partially observable environments. Partial observation often leads to the model hallucinating the planning. Therefore, our diffusion-based method jointly models both state trajectory and goal estimation to improve the reliability of the generated plan, given the limited available information at each step. To better leverage newly discovered information along the plan execution for a higher success rate, we propose an on-the-fly planning algorithm to collaborate with the diffusion-based planner. The proposed framework achieves promising performances in various embodied AI tasks, including vision-language navigation, object manipulation, and task planning in a photorealistic virtual environment. The code is available at: https://github.com/joeyy5588/planning-as-inpainting.","url_abs":"https://arxiv.org/abs/2312.01097v1","url_pdf":"https://arxiv.org/pdf/2312.01097v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"planning-as-in-painting-a-diffusion-based","repo_url":"https://github.com/joeyy5588/planning-as-inpainting","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"task-planning","task_name":"Task Planning"},{"task_slug":"vision-language-navigation","task_name":"Vision-Language Navigation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2312.01097","atlas_url":"https://app.syntology.ai/?focus=2312.01097","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.01097"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/joeyy5588/planning-as-inpainting","reach":{"status":"ok"}}],"summary":{"ran":6,"unverified":5},"by_repo_kind":{"official":{"samples":11,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"1c5d7c6c8e55f6c1","entry":"apply_kernel_at_target","repo":"joeyy5588/planning-as-inpainting","repo_kind":"official","path":"datasets/alfred.py","file_url":"https://github.com/joeyy5588/planning-as-inpainting/blob/HEAD/datasets/alfred.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1c5d7c6c8e55f6c1"}},{"code_sha256_prefix":"037f4a90e9d91337","entry":"astar","repo":"joeyy5588/planning-as-inpainting","repo_kind":"official","path":"datasets/astar.py","file_url":"https://github.com/joeyy5588/planning-as-inpainting/blob/HEAD/datasets/astar.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"037f4a90e9d91337"}},{"code_sha256_prefix":"76118b3f3c3ba26a","entry":"blur_target_with_kernel","repo":"joeyy5588/planning-as-inpainting","repo_kind":"official","path":"datasets/alfred.py","file_url":"https://github.com/joeyy5588/planning-as-inpainting/blob/HEAD/datasets/alfred.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"76118b3f3c3ba26a"}},{"code_sha256_prefix":"40966c231d479773","entry":"gaussian_kernel","repo":"joeyy5588/planning-as-inpainting","repo_kind":"official","path":"datasets/alfred.py","file_url":"https://github.com/joeyy5588/planning-as-inpainting/blob/HEAD/datasets/alfred.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"40966c231d479773"}},{"code_sha256_prefix":"ee3a6cf6057e204e","entry":"gaussian_kernel","repo":"joeyy5588/planning-as-inpainting","repo_kind":"official","path":"datasets/gridworld_heatmap.py","file_url":"https://github.com/joeyy5588/planning-as-inpainting/blob/HEAD/datasets/gridworld_heatmap.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ee3a6cf6057e204e"}},{"code_sha256_prefix":"6ceb7088925eb34f","entry":"get_action","repo":"joeyy5588/planning-as-inpainting","repo_kind":"official","path":"datasets/gridworld.py","file_url":"https://github.com/joeyy5588/planning-as-inpainting/blob/HEAD/datasets/gridworld.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6ceb7088925eb34f"}},{"code_sha256_prefix":"8fa39392f6f64629","entry":"generate_new_path","repo":"joeyy5588/planning-as-inpainting","repo_kind":"official","path":"datasets/gridworld.py","file_url":"https://github.com/joeyy5588/planning-as-inpainting/blob/HEAD/datasets/gridworld.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8fa39392f6f64629"}},{"code_sha256_prefix":"8a01e6b0df69de62","entry":"generate_new_path","repo":"joeyy5588/planning-as-inpainting","repo_kind":"official","path":"datasets/gridworld_2d.py","file_url":"https://github.com/joeyy5588/planning-as-inpainting/blob/HEAD/datasets/gridworld_2d.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8a01e6b0df69de62"}},{"code_sha256_prefix":"fce95f3d14e45577","entry":"generate_new_path","repo":"joeyy5588/planning-as-inpainting","repo_kind":"official","path":"datasets/gridworld_condition.py","file_url":"https://github.com/joeyy5588/planning-as-inpainting/blob/HEAD/datasets/gridworld_condition.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fce95f3d14e45577"}},{"code_sha256_prefix":"6b0160d08f910e04","entry":"generate_new_path","repo":"joeyy5588/planning-as-inpainting","repo_kind":"official","path":"datasets/gridworld_repr.py","file_url":"https://github.com/joeyy5588/planning-as-inpainting/blob/HEAD/datasets/gridworld_repr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6b0160d08f910e04"}},{"code_sha256_prefix":"eedf2e53d640351f","entry":"generate_new_path_2","repo":"joeyy5588/planning-as-inpainting","repo_kind":"official","path":"datasets/gridworld_2d.py","file_url":"https://github.com/joeyy5588/planning-as-inpainting/blob/HEAD/datasets/gridworld_2d.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eedf2e53d640351f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}