{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-robust-offline-to-online","title":"Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness","arxiv_id":"2309.16973","date":"2023-09-29","proceeding":null,"authors":["Xiaoyu Wen","Xudong Yu","Rui Yang","HaoYuan Chen","Chenjia Bai","Zhen Wang"],"abstract":"To obtain a near-optimal policy with fewer interactions in Reinforcement Learning (RL), a promising approach involves the combination of offline RL, which enhances sample efficiency by leveraging offline datasets, and online RL, which explores informative transitions by interacting with the environment. Offline-to-Online (O2O) RL provides a paradigm for improving an offline trained agent within limited online interactions. However, due to the significant distribution shift between online experiences and offline data, most offline RL algorithms suffer from performance drops and fail to achieve stable policy improvement in O2O adaptation. To address this problem, we propose the Robust Offline-to-Online (RO2O) algorithm, designed to enhance offline policies through uncertainty and smoothness, and to mitigate the performance drop in online adaptation. Specifically, RO2O incorporates Q-ensemble for uncertainty penalty and adversarial samples for policy and value smoothness, which enable RO2O to maintain a consistent learning procedure in online adaptation without requiring special changes to the learning objective. Theoretical analyses in linear MDPs demonstrate that the uncertainty and smoothness lead to a tighter optimality bound in O2O against distribution shift. Experimental results illustrate the superiority of RO2O in facilitating stable offline-to-online learning and achieving significant improvement with limited online interactions.","url_abs":"https://arxiv.org/abs/2309.16973v2","url_pdf":"https://arxiv.org/pdf/2309.16973v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-robust-offline-to-online","repo_url":"https://github.com/battlewen/ro2o","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"offline-rl","task_name":"Offline RL"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2309.16973","atlas_url":"https://app.syntology.ai/?focus=2309.16973","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.16973"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/battlewen/ro2o","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":2,"ran":1,"ran_honours":2,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"4e08ac58e3891cfb","entry":"compute_mean_std","repo":"battlewen/ro2o","repo_kind":"official","path":"algorithms/finetune/awac.py","file_url":"https://github.com/battlewen/ro2o/blob/HEAD/algorithms/finetune/awac.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4e08ac58e3891cfb"}},{"code_sha256_prefix":"639b4986c270df90","entry":"compute_mean_std","repo":"battlewen/ro2o","repo_kind":"official","path":"algorithms/finetune/iql.py","file_url":"https://github.com/battlewen/ro2o/blob/HEAD/algorithms/finetune/iql.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"639b4986c270df90"}},{"code_sha256_prefix":"26dee61e902bf417","entry":"eval_actor","repo":"battlewen/ro2o","repo_kind":"official","path":"algorithms/finetune/ro2o_ft_ant_LCB.py","file_url":"https://github.com/battlewen/ro2o/blob/HEAD/algorithms/finetune/ro2o_ft_ant_LCB.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"26dee61e902bf417"}},{"code_sha256_prefix":"ba80f9ff81b1de5f","entry":"normalize_states","repo":"battlewen/ro2o","repo_kind":"official","path":"algorithms/finetune/awac.py","file_url":"https://github.com/battlewen/ro2o/blob/HEAD/algorithms/finetune/awac.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ba80f9ff81b1de5f"}},{"code_sha256_prefix":"136a4348dc060901","entry":"return_reward_range","repo":"battlewen/ro2o","repo_kind":"official","path":"algorithms/finetune/ro2o_ft_ant_LCB.py","file_url":"https://github.com/battlewen/ro2o/blob/HEAD/algorithms/finetune/ro2o_ft_ant_LCB.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"136a4348dc060901"}},{"code_sha256_prefix":"4dd56ecf0f797666","entry":"wrap_env","repo":"battlewen/ro2o","repo_kind":"official","path":"algorithms/finetune/ro2o_ft_ant_LCB.py","file_url":"https://github.com/battlewen/ro2o/blob/HEAD/algorithms/finetune/ro2o_ft_ant_LCB.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4dd56ecf0f797666"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}