{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/large-scale-actionless-video-pre-training-via","title":"Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training","arxiv_id":"2402.14407","date":"2024-02-22","proceeding":null,"authors":["Haoran He","Chenjia Bai","Ling Pan","Weinan Zhang","Bin Zhao","Xuelong Li"],"abstract":"Learning a generalist embodied agent capable of completing multiple tasks poses challenges, primarily stemming from the scarcity of action-labeled robotic datasets. In contrast, a vast amount of human videos exist, capturing intricate tasks and interactions with the physical world. Promising prospects arise for utilizing actionless human videos for pre-training and transferring the knowledge to facilitate robot policy learning through limited robot demonstrations. However, it remains a challenge due to the domain gap between humans and robots. Moreover, it is difficult to extract useful information representing the dynamic world from human videos, because of its noisy and multimodal data structure. In this paper, we introduce a novel framework to tackle these challenges, which leverages a unified discrete diffusion to combine generative pre-training on human videos and policy fine-tuning on a small number of action-labeled robot videos. We start by compressing both human and robot videos into unified video tokens. In the pre-training stage, we employ a discrete diffusion model with a mask-and-replace diffusion strategy to predict future video tokens in the latent space. In the fine-tuning stage, we harness the imagined future videos to guide low-level action learning with a limited set of robot data. Experiments demonstrate that our method generates high-fidelity future videos for planning and enhances the fine-tuned policies compared to previous state-of-the-art approaches with superior performance. Our project website is available at https://video-diff.github.io/.","url_abs":"https://arxiv.org/abs/2402.14407v4","url_pdf":"https://arxiv.org/pdf/2402.14407v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"large-scale-actionless-video-pre-training-via","repo_url":"https://github.com/tinnerhrhe/VPDD","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null}],"tasks":[],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.14407","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.14407"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/peract/peract","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/RLAgent/factor-world","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tinnerhrhe/VPDD","reach":null}],"summary":{"ran":13,"ran_draft_wrong":2,"ran_violates":2,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1},"found_in_text":{"samples":12,"ran":12,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e163906c3cdc2aa6","entry":"Huber","repo":"tinnerhrhe/VPDD","repo_kind":"official","path":"diffuser/models/diffusion.py","file_url":"https://github.com/tinnerhrhe/VPDD/blob/HEAD/diffuser/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e163906c3cdc2aa6"}},{"code_sha256_prefix":"acf269bb622e8d6a","entry":"StateHuber","repo":"tinnerhrhe/VPDD","repo_kind":"official","path":"diffuser/models/diffusion.py","file_url":"https://github.com/tinnerhrhe/VPDD/blob/HEAD/diffuser/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"acf269bb622e8d6a"}},{"code_sha256_prefix":"43177911c1fd1d18","entry":"WeightedStateLoss","repo":"tinnerhrhe/VPDD","repo_kind":"official","path":"diffuser/models/diffusion.py","file_url":"https://github.com/tinnerhrhe/VPDD/blob/HEAD/diffuser/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"43177911c1fd1d18"}},{"code_sha256_prefix":"c99fb7b9654ea364","entry":"act_layer","repo":"peract/peract","repo_kind":"found_in_text","path":"helpers/network_utils.py","file_url":"https://github.com/peract/peract/blob/HEAD/helpers/network_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c99fb7b9654ea364"}},{"code_sha256_prefix":"8458f9ba0361ff43","entry":"cache_fn","repo":"peract/peract","repo_kind":"found_in_text","path":"agents/peract_bc/perceiver_lang_io.py","file_url":"https://github.com/peract/peract/blob/HEAD/agents/peract_bc/perceiver_lang_io.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8458f9ba0361ff43"}},{"code_sha256_prefix":"60fff7c3c400d7ff","entry":"default","repo":"peract/peract","repo_kind":"found_in_text","path":"agents/peract_bc/perceiver_lang_io.py","file_url":"https://github.com/peract/peract/blob/HEAD/agents/peract_bc/perceiver_lang_io.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"60fff7c3c400d7ff"}},{"code_sha256_prefix":"8d04d499d75080d0","entry":"default_sample_fn","repo":"tinnerhrhe/VPDD","repo_kind":"official","path":"diffuser/models/diffusion.py","file_url":"https://github.com/tinnerhrhe/VPDD/blob/HEAD/diffuser/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8d04d499d75080d0"}},{"code_sha256_prefix":"051af9d2e2264347","entry":"dict_to_list_of_dicts","repo":"RLAgent/factor-world","repo_kind":"found_in_text","path":"envs/factors/factor_wrapper.py","file_url":"https://github.com/RLAgent/factor-world/blob/HEAD/envs/factors/factor_wrapper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"051af9d2e2264347"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"peract/peract","repo_kind":"found_in_text","path":"agents/peract_bc/perceiver_lang_io.py","file_url":"https://github.com/peract/peract/blob/HEAD/agents/peract_bc/perceiver_lang_io.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"0a61ee3eb30e145f","entry":"find_minimum_difference","repo":"peract/peract","repo_kind":"found_in_text","path":"helpers/demo_loading_utils.py","file_url":"https://github.com/peract/peract/blob/HEAD/helpers/demo_loading_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0a61ee3eb30e145f"}},{"code_sha256_prefix":"5b55e1bed13f5a58","entry":"full_assets_path_for","repo":"RLAgent/factor-world","repo_kind":"found_in_text","path":"envs/asset_path_utils.py","file_url":"https://github.com/RLAgent/factor-world/blob/HEAD/envs/asset_path_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5b55e1bed13f5a58"}},{"code_sha256_prefix":"cda51c68aff53692","entry":"full_mso_path_for","repo":"RLAgent/factor-world","repo_kind":"found_in_text","path":"envs/asset_path_utils.py","file_url":"https://github.com/RLAgent/factor-world/blob/HEAD/envs/asset_path_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cda51c68aff53692"}},{"code_sha256_prefix":"755db90b1e706d42","entry":"full_v2_path_for","repo":"RLAgent/factor-world","repo_kind":"found_in_text","path":"envs/asset_path_utils.py","file_url":"https://github.com/RLAgent/factor-world/blob/HEAD/envs/asset_path_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"755db90b1e706d42"}},{"code_sha256_prefix":"9ed9933be23b28a5","entry":"norm_layer1d","repo":"peract/peract","repo_kind":"found_in_text","path":"helpers/network_utils.py","file_url":"https://github.com/peract/peract/blob/HEAD/helpers/network_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9ed9933be23b28a5"}},{"code_sha256_prefix":"d0264b5831943fb9","entry":"norm_layer2d","repo":"peract/peract","repo_kind":"found_in_text","path":"helpers/network_utils.py","file_url":"https://github.com/peract/peract/blob/HEAD/helpers/network_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d0264b5831943fb9"}},{"code_sha256_prefix":"295aa3b49a6888ac","entry":"sample_without_replacement","repo":"RLAgent/factor-world","repo_kind":"found_in_text","path":"envs/factors/factor_wrapper.py","file_url":"https://github.com/RLAgent/factor-world/blob/HEAD/envs/factors/factor_wrapper.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"295aa3b49a6888ac"}},{"code_sha256_prefix":"ee809dc4a6634822","entry":"videoloss","repo":"tinnerhrhe/VPDD","repo_kind":"official","path":"diffuser/models/diffusion.py","file_url":"https://github.com/tinnerhrhe/VPDD/blob/HEAD/diffuser/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ee809dc4a6634822"}},{"code_sha256_prefix":"81f1f429fb4a7f3e","entry":"DTDiffusion","repo":"tinnerhrhe/VPDD","repo_kind":"official","path":"diffuser/models/diffusion.py","file_url":"https://github.com/tinnerhrhe/VPDD/blob/HEAD/diffuser/models/diffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"81f1f429fb4a7f3e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}