{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/peac-unsupervised-pre-training-for-cross","title":"PEAC: Unsupervised Pre-training for Cross-Embodiment Reinforcement Learning","arxiv_id":"2405.14073","date":"2024-05-23","proceeding":null,"authors":["Chengyang Ying","Zhongkai Hao","Xinning Zhou","Xuezhou Xu","Hang Su","Xingxing Zhang","Jun Zhu"],"abstract":"Designing generalizable agents capable of adapting to diverse embodiments has achieved significant attention in Reinforcement Learning (RL), which is critical for deploying RL agents in various real-world applications. Previous Cross-Embodiment RL approaches have focused on transferring knowledge across embodiments within specific tasks. These methods often result in knowledge tightly coupled with those tasks and fail to adequately capture the distinct characteristics of different embodiments. To address this limitation, we introduce the notion of Cross-Embodiment Unsupervised RL (CEURL), which leverages unsupervised learning to enable agents to acquire embodiment-aware and task-agnostic knowledge through online interactions within reward-free environments. We formulate CEURL as a novel Controlled Embodiment Markov Decision Process (CE-MDP) and systematically analyze CEURL's pre-training objectives under CE-MDP. Based on these analyses, we develop a novel algorithm Pre-trained Embodiment-Aware Control (PEAC) for handling CEURL, incorporating an intrinsic reward function specifically designed for cross-embodiment pre-training. PEAC not only provides an intuitive optimization strategy for cross-embodiment pre-training but also can integrate flexibly with existing unsupervised RL methods, facilitating cross-embodiment exploration and skill discovery. Extensive experiments in both simulated (e.g., DMC and Robosuite) and real-world environments (e.g., legged locomotion) demonstrate that PEAC significantly improves adaptation performance and cross-embodiment generalization, demonstrating its effectiveness in overcoming the unique challenges of CEURL. The project page and code are in https://yingchengyang.github.io/ceurl.","url_abs":"https://arxiv.org/abs/2405.14073v2","url_pdf":"https://arxiv.org/pdf/2405.14073v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"peac-unsupervised-pre-training-for-cross","repo_url":"https://github.com/thu-ml/CEURL","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"unsupervised-pre-training","task_name":"Unsupervised Pre-training"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.14073","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.14073"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thu-ml/CEURL","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"33825bfaf6a0b7fc","entry":"count_episodes","repo":"thu-ml/CEURL","repo_kind":"official","path":"DMC_image/dreamer_replay.py","file_url":"https://github.com/thu-ml/CEURL/blob/HEAD/DMC_image/dreamer_replay.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"33825bfaf6a0b7fc"}},{"code_sha256_prefix":"2478bb95b38ef24f","entry":"grad_norm","repo":"thu-ml/CEURL","repo_kind":"official","path":"DMC_image/utils.py","file_url":"https://github.com/thu-ml/CEURL/blob/HEAD/DMC_image/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2478bb95b38ef24f"}},{"code_sha256_prefix":"d33901ea1fb03092","entry":"load_episodes","repo":"thu-ml/CEURL","repo_kind":"official","path":"DMC_image/dreamer_replay.py","file_url":"https://github.com/thu-ml/CEURL/blob/HEAD/DMC_image/dreamer_replay.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d33901ea1fb03092"}},{"code_sha256_prefix":"55a2143c42c8051e","entry":"make_dreamer_agent","repo":"thu-ml/CEURL","repo_kind":"official","path":"DMC_image/dreamer_finetune.py","file_url":"https://github.com/thu-ml/CEURL/blob/HEAD/DMC_image/dreamer_finetune.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"55a2143c42c8051e"}},{"code_sha256_prefix":"8708aaca52006e47","entry":"param_norm","repo":"thu-ml/CEURL","repo_kind":"official","path":"DMC_image/utils.py","file_url":"https://github.com/thu-ml/CEURL/blob/HEAD/DMC_image/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8708aaca52006e47"}},{"code_sha256_prefix":"6244e9922f4e0610","entry":"to_torch","repo":"thu-ml/CEURL","repo_kind":"official","path":"DMC_image/utils.py","file_url":"https://github.com/thu-ml/CEURL/blob/HEAD/DMC_image/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6244e9922f4e0610"}},{"code_sha256_prefix":"2a76dfe317adeb03","entry":"make_agent","repo":"thu-ml/CEURL","repo_kind":"official","path":"DMC_image/dreamer_finetune.py","file_url":"https://github.com/thu-ml/CEURL/blob/HEAD/DMC_image/dreamer_finetune.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2a76dfe317adeb03"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}