{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rorl-robust-offline-reinforcement-learning","title":"RORL: Robust Offline Reinforcement Learning via Conservative Smoothing","arxiv_id":"2206.02829","date":"2022-06-06","proceeding":null,"authors":["Rui Yang","Chenjia Bai","Xiaoteng Ma","Zhaoran Wang","Chongjie Zhang","Lei Han"],"abstract":"Offline reinforcement learning (RL) provides a promising direction to exploit massive amount of offline data for complex decision-making tasks. Due to the distribution shift issue, current offline RL algorithms are generally designed to be conservative in value estimation and action selection. However, such conservatism can impair the robustness of learned policies when encountering observation deviation under realistic conditions, such as sensor errors and adversarial attacks. To trade off robustness and conservatism, we propose Robust Offline Reinforcement Learning (RORL) with a novel conservative smoothing technique. In RORL, we explicitly introduce regularization on the policy and the value function for states near the dataset, as well as additional conservative value estimation on these states. Theoretically, we show RORL enjoys a tighter suboptimality bound than recent theoretical results in linear MDPs. We demonstrate that RORL can achieve state-of-the-art performance on the general offline RL benchmark and is considerably robust to adversarial observation perturbations.","url_abs":"https://arxiv.org/abs/2206.02829v3","url_pdf":"https://arxiv.org/pdf/2206.02829v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rorl-robust-offline-reinforcement-learning","repo_url":"https://github.com/yangrui2015/rorl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"offline-rl","task_name":"Offline RL"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2206.02829","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.02829"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yangrui2015/rorl","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"75bf04014baa9e16","entry":"get_obs_mean_std","repo":"yangrui2015/rorl","repo_kind":"official","path":"attackers/data_mean_std.py","file_url":"https://github.com/yangrui2015/rorl/blob/HEAD/attackers/data_mean_std.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"75bf04014baa9e16"}},{"code_sha256_prefix":"c8d4b850e6428750","entry":"get_policy_kl","repo":"yangrui2015/rorl","repo_kind":"official","path":"attackers/attacker.py","file_url":"https://github.com/yangrui2015/rorl/blob/HEAD/attackers/attacker.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c8d4b850e6428750"}},{"code_sha256_prefix":"159455d9b3af1142","entry":"load_dataset","repo":"yangrui2015/rorl","repo_kind":"official","path":"experiment_utils/utils.py","file_url":"https://github.com/yangrui2015/rorl/blob/HEAD/experiment_utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"159455d9b3af1142"}},{"code_sha256_prefix":"a31ef623997c87b7","entry":"load_exp_data","repo":"yangrui2015/rorl","repo_kind":"official","path":"experiment_utils/utils.py","file_url":"https://github.com/yangrui2015/rorl/blob/HEAD/experiment_utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a31ef623997c87b7"}},{"code_sha256_prefix":"c0cd96043462cd86","entry":"load_json","repo":"yangrui2015/rorl","repo_kind":"official","path":"experiment_utils/utils.py","file_url":"https://github.com/yangrui2015/rorl/blob/HEAD/experiment_utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c0cd96043462cd86"}},{"code_sha256_prefix":"de96789724b92f83","entry":"optimize_para","repo":"yangrui2015/rorl","repo_kind":"official","path":"attackers/attacker.py","file_url":"https://github.com/yangrui2015/rorl/blob/HEAD/attackers/attacker.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"de96789724b92f83"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}