{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/offline-retraining-for-online-rl-decoupled","title":"Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias","arxiv_id":"2310.08558","date":"2023-10-12","proceeding":null,"authors":["Max Sobol Mark","Archit Sharma","Fahim Tajwar","Rafael Rafailov","Sergey Levine","Chelsea Finn"],"abstract":"It is desirable for policies to optimistically explore new states and behaviors during online reinforcement learning (RL) or fine-tuning, especially when prior offline data does not provide enough state coverage. However, exploration bonuses can bias the learned policy, and our experiments find that naive, yet standard use of such bonuses can fail to recover a performant policy. Concurrently, pessimistic training in offline RL has enabled recovery of performant policies from static datasets. Can we leverage offline RL to recover better policies from online interaction? We make a simple observation that a policy can be trained from scratch on all interaction data with pessimistic objectives, thereby decoupling the policies used for data collection and for evaluation. Specifically, we propose offline retraining, a policy extraction step at the end of online fine-tuning in our Offline-to-Online-to-Offline (OOO) framework for reinforcement learning (RL). An optimistic (exploration) policy is used to interact with the environment, and a separate pessimistic (exploitation) policy is trained on all the observed data for evaluation. Such decoupling can reduce any bias from online interaction (intrinsic rewards, primacy bias) in the evaluation policy, and can allow more exploratory behaviors during online interaction which in turn can generate better data for exploitation. OOO is complementary to several offline-to-online RL and online RL methods, and improves their average performance by 14% to 26% in our fine-tuning experiments, achieves state-of-the-art performance on several environments in the D4RL benchmarks, and improves online RL performance by 165% on two OpenAI gym environments. Further, OOO can enable fine-tuning from incomplete offline datasets where prior methods can fail to recover a performant policy. Implementation: https://github.com/MaxSobolMark/OOO","url_abs":"https://arxiv.org/abs/2310.08558v1","url_pdf":"https://arxiv.org/pdf/2310.08558v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"offline-retraining-for-online-rl-decoupled","repo_url":"https://github.com/MaxSobolMark/OOO","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}}],"tasks":[{"task_slug":"d4rl","task_name":"D4RL"},{"task_slug":"offline-rl","task_name":"Offline RL"},{"task_slug":"openai-gym","task_name":"OpenAI Gym"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.08558","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.08558"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MaxSobolMark/OOO","reach":{"status":"ok"}}],"summary":{"ran":5,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":9,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"42bdf6d9cb0aaad5","entry":"calculate_intrinsic_reward","repo":"MaxSobolMark/OOO","repo_kind":"official","path":"OOO_for_calql/rnd_net.py","file_url":"https://github.com/MaxSobolMark/OOO/blob/HEAD/OOO_for_calql/rnd_net.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"42bdf6d9cb0aaad5"}},{"code_sha256_prefix":"7a2c5b4e604fd307","entry":"combine","repo":"MaxSobolMark/OOO","repo_kind":"official","path":"OOO_for_rlpd/train_finetuning_decoupled.py","file_url":"https://github.com/MaxSobolMark/OOO/blob/HEAD/OOO_for_rlpd/train_finetuning_decoupled.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7a2c5b4e604fd307"}},{"code_sha256_prefix":"d2f199c948f2a9ed","entry":"extend_and_repeat","repo":"MaxSobolMark/OOO","repo_kind":"official","path":"OOO_for_calql/jax_utils.py","file_url":"https://github.com/MaxSobolMark/OOO/blob/HEAD/OOO_for_calql/jax_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d2f199c948f2a9ed"}},{"code_sha256_prefix":"1d46473b7c1c2186","entry":"functional_running_mean_std_update","repo":"MaxSobolMark/OOO","repo_kind":"official","path":"OOO_for_calql/rnd_net.py","file_url":"https://github.com/MaxSobolMark/OOO/blob/HEAD/OOO_for_calql/rnd_net.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1d46473b7c1c2186"}},{"code_sha256_prefix":"cc35af8b18b7dc58","entry":"mse_loss","repo":"MaxSobolMark/OOO","repo_kind":"official","path":"OOO_for_calql/jax_utils.py","file_url":"https://github.com/MaxSobolMark/OOO/blob/HEAD/OOO_for_calql/jax_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cc35af8b18b7dc58"}},{"code_sha256_prefix":"d78e83e4520027b1","entry":"wrap_function_with_rng","repo":"MaxSobolMark/OOO","repo_kind":"official","path":"OOO_for_calql/jax_utils.py","file_url":"https://github.com/MaxSobolMark/OOO/blob/HEAD/OOO_for_calql/jax_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d78e83e4520027b1"}},{"code_sha256_prefix":"bc616c63f1ef3dc2","entry":"get_hand_dataset_with_mc_calculation","repo":"MaxSobolMark/OOO","repo_kind":"official","path":"OOO_for_calql/replay_buffer.py","file_url":"https://github.com/MaxSobolMark/OOO/blob/HEAD/OOO_for_calql/replay_buffer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bc616c63f1ef3dc2"}},{"code_sha256_prefix":"1a78f3efb730aee9","entry":"update_rnd","repo":"MaxSobolMark/OOO","repo_kind":"official","path":"OOO_for_calql/rnd_net.py","file_url":"https://github.com/MaxSobolMark/OOO/blob/HEAD/OOO_for_calql/rnd_net.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1a78f3efb730aee9"}},{"code_sha256_prefix":"f23c0c9faf00d830","entry":"update_target_network","repo":"MaxSobolMark/OOO","repo_kind":"official","path":"OOO_for_calql/model.py","file_url":"https://github.com/MaxSobolMark/OOO/blob/HEAD/OOO_for_calql/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f23c0c9faf00d830"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}