{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/learning-to-act-from-actionless-videos","title":"Learning to Act from Actionless Videos through Dense Correspondences","arxiv_id":"2310.08576","date":"2023-10-12","proceeding":null,"authors":["Po-Chen Ko","Jiayuan Mao","Yilun Du","Shao-Hua Sun","Joshua B. Tenenbaum"],"abstract":"In this work, we present an approach to construct a video-based robot policy capable of reliably executing diverse tasks across different robots and environments from few video demonstrations without using any action annotations. Our method leverages images as a task-agnostic representation, encoding both the state and action information, and text as a general representation for specifying robot goals. By synthesizing videos that ``hallucinate'' robot executing actions and in combination with dense correspondences between frames, our approach can infer the closed-formed action to execute to an environment without the need of any explicit action labels. This unique capability allows us to train the policy solely based on RGB videos and deploy learned policies to various robotic tasks. We demonstrate the efficacy of our approach in learning policies on table-top manipulation and navigation tasks. Additionally, we contribute an open-source framework for efficient video modeling, enabling the training of high-fidelity policy models with four GPUs within a single day.","url_abs":"https://arxiv.org/abs/2310.08576v1","url_pdf":"https://arxiv.org/pdf/2310.08576v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"learning-to-act-from-actionless-videos","repo_url":"https://github.com/flow-diffusion/avdc","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"learning-to-act-from-actionless-videos","repo_url":"https://github.com/video-as-agent/videoagent","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.08576","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.08576"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/flow-diffusion/avdc","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/video-as-agent/videoagent","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":1,"ran_violates":4,"ran":3,"unverified":1},"by_repo_kind":{"listed":{"samples":9,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7f1040f5e3991d5e","entry":"identity","repo":"flow-diffusion/avdc","repo_kind":"listed","path":"flowdiffusion/model/imagen.py","file_url":"https://github.com/flow-diffusion/avdc/blob/HEAD/flowdiffusion/model/imagen.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":2,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7f1040f5e3991d5e"}},{"code_sha256_prefix":"d1ef6b8cb9a28a53","entry":"default","repo":"flow-diffusion/avdc","repo_kind":"listed","path":"flowdiffusion/goal_diffusion.py","file_url":"https://github.com/flow-diffusion/avdc/blob/HEAD/flowdiffusion/goal_diffusion.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d1ef6b8cb9a28a53"}},{"code_sha256_prefix":"f46bcb6099334fa6","entry":"downsample_2d","repo":"flow-diffusion/avdc","repo_kind":"listed","path":"flowdiffusion/model/resnet.py","file_url":"https://github.com/flow-diffusion/avdc/blob/HEAD/flowdiffusion/model/resnet.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f46bcb6099334fa6"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"flow-diffusion/avdc","repo_kind":"listed","path":"flowdiffusion/model/imagen.py","file_url":"https://github.com/flow-diffusion/avdc/blob/HEAD/flowdiffusion/model/imagen.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"608e364a9d2376a3","entry":"exists","repo":"flow-diffusion/avdc","repo_kind":"listed","path":"flowdiffusion/goal_diffusion.py","file_url":"https://github.com/flow-diffusion/avdc/blob/HEAD/flowdiffusion/goal_diffusion.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"608e364a9d2376a3"}},{"code_sha256_prefix":"3072294c9b32cf4b","entry":"first","repo":"flow-diffusion/avdc","repo_kind":"listed","path":"flowdiffusion/model/imagen.py","file_url":"https://github.com/flow-diffusion/avdc/blob/HEAD/flowdiffusion/model/imagen.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3072294c9b32cf4b"}},{"code_sha256_prefix":"a191f9f21d279753","entry":"rearrange_dims","repo":"flow-diffusion/avdc","repo_kind":"listed","path":"flowdiffusion/model/resnet.py","file_url":"https://github.com/flow-diffusion/avdc/blob/HEAD/flowdiffusion/model/resnet.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a191f9f21d279753"}},{"code_sha256_prefix":"f67b30f563b85481","entry":"tensors2vectors","repo":"flow-diffusion/avdc","repo_kind":"listed","path":"flowdiffusion/goal_diffusion.py","file_url":"https://github.com/flow-diffusion/avdc/blob/HEAD/flowdiffusion/goal_diffusion.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f67b30f563b85481"}},{"code_sha256_prefix":"31e7ed79c46ee9a9","entry":"upsample_2d","repo":"flow-diffusion/avdc","repo_kind":"listed","path":"flowdiffusion/model/resnet.py","file_url":"https://github.com/flow-diffusion/avdc/blob/HEAD/flowdiffusion/model/resnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"31e7ed79c46ee9a9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}