{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/uni-o4-unifying-online-and-offline-deep","title":"Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization","arxiv_id":"2311.03351","date":"2023-11-06","proceeding":null,"authors":["Kun Lei","Zhengmao He","Chenhao Lu","Kaizhe Hu","Yang Gao","Huazhe Xu"],"abstract":"Combining offline and online reinforcement learning (RL) is crucial for efficient and safe learning. However, previous approaches treat offline and online learning as separate procedures, resulting in redundant designs and limited performance. We ask: Can we achieve straightforward yet effective offline and online learning without introducing extra conservatism or regularization? In this study, we propose Uni-o4, which utilizes an on-policy objective for both offline and online learning. Owning to the alignment of objectives in two phases, the RL agent can transfer between offline and online learning seamlessly. This property enhances the flexibility of the learning paradigm, allowing for arbitrary combinations of pretraining, fine-tuning, offline, and online learning. In the offline phase, specifically, Uni-o4 leverages diverse ensemble policies to address the mismatch issues between the estimated behavior policy and the offline dataset. Through a simple offline policy evaluation (OPE) approach, Uni-o4 can achieve multi-step policy improvement safely. We demonstrate that by employing the method above, the fusion of these two paradigms can yield superior offline initialization as well as stable and rapid online fine-tuning capabilities. Through real-world robot tasks, we highlight the benefits of this paradigm for rapid deployment in challenging, previously unseen real-world environments. Additionally, through comprehensive evaluations using numerous simulated benchmarks, we substantiate that our method achieves state-of-the-art performance in both offline and offline-to-online fine-tuning learning. Our website: https://lei-kun.github.io/uni-o4/ .","url_abs":"https://arxiv.org/abs/2311.03351v4","url_pdf":"https://arxiv.org/pdf/2311.03351v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"uni-o4-unifying-online-and-offline-deep","repo_url":"https://github.com/Lei-Kun/Uni-O4","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2311.03351","atlas_url":"https://app.syntology.ai/?focus=2311.03351","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.03351"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Lei-Kun/Uni-O4","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"ran_honours":1},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"55423074898a45ac","entry":"MLP","repo":"Lei-Kun/Uni-O4","repo_kind":"official","path":"net.py","file_url":"https://github.com/Lei-Kun/Uni-O4/blob/HEAD/net.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"55423074898a45ac"}},{"code_sha256_prefix":"010405be9f974eca","entry":"dynamics_eval","repo":"Lei-Kun/Uni-O4","repo_kind":"official","path":"dynamics_eval.py","file_url":"https://github.com/Lei-Kun/Uni-O4/blob/HEAD/dynamics_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"010405be9f974eca"}},{"code_sha256_prefix":"1c0b96eb357d508b","entry":"rollout","repo":"Lei-Kun/Uni-O4","repo_kind":"official","path":"dynamics_eval.py","file_url":"https://github.com/Lei-Kun/Uni-O4/blob/HEAD/dynamics_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1c0b96eb357d508b"}},{"code_sha256_prefix":"c8dc65736e6bef1e","entry":"soft_clamp","repo":"Lei-Kun/Uni-O4","repo_kind":"official","path":"net.py","file_url":"https://github.com/Lei-Kun/Uni-O4/blob/HEAD/net.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c8dc65736e6bef1e"}},{"code_sha256_prefix":"109c5738b44665bb","entry":"soft_clamp","repo":"Lei-Kun/Uni-O4","repo_kind":"official","path":"ppo_finetune/net.py","file_url":"https://github.com/Lei-Kun/Uni-O4/blob/HEAD/ppo_finetune/net.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"109c5738b44665bb"}},{"code_sha256_prefix":"3ebb420a5d10cc64","entry":"update_mean_var_count_from_moments","repo":"Lei-Kun/Uni-O4","repo_kind":"official","path":"ppo_finetune/normalization.py","file_url":"https://github.com/Lei-Kun/Uni-O4/blob/HEAD/ppo_finetune/normalization.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3ebb420a5d10cc64"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}