{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-robust-offline-reinforcement-learning","title":"Towards Robust Offline Reinforcement Learning under Diverse Data Corruption","arxiv_id":"2310.12955","date":"2023-10-19","proceeding":null,"authors":["Rui Yang","Han Zhong","Jiawei Xu","Amy Zhang","Chongjie Zhang","Lei Han","Tong Zhang"],"abstract":"Offline reinforcement learning (RL) presents a promising approach for learning reinforced policies from offline datasets without the need for costly or unsafe interactions with the environment. However, datasets collected by humans in real-world environments are often noisy and may even be maliciously corrupted, which can significantly degrade the performance of offline RL. In this work, we first investigate the performance of current offline RL algorithms under comprehensive data corruption, including states, actions, rewards, and dynamics. Our extensive experiments reveal that implicit Q-learning (IQL) demonstrates remarkable resilience to data corruption among various offline RL algorithms. Furthermore, we conduct both empirical and theoretical analyses to understand IQL's robust performance, identifying its supervised policy learning scheme as the key factor. Despite its relative robustness, IQL still suffers from heavy-tail targets of Q functions under dynamics corruption. To tackle this challenge, we draw inspiration from robust statistics to employ the Huber loss to handle the heavy-tailedness and utilize quantile estimators to balance penalization for corrupted data and learning stability. By incorporating these simple yet effective modifications into IQL, we propose a more robust offline RL approach named Robust IQL (RIQL). Extensive experiments demonstrate that RIQL exhibits highly robust performance when subjected to diverse data corruption scenarios.","url_abs":"https://arxiv.org/abs/2310.12955v3","url_pdf":"https://arxiv.org/pdf/2310.12955v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-robust-offline-reinforcement-learning","repo_url":"https://github.com/yangrui2015/riql","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"towards-robust-offline-reinforcement-learning","repo_url":"https://github.com/zzmtsvv/ORL","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"offline-rl","task_name":"Offline RL"},{"task_slug":"q-learning","task_name":"Q-Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"huber-loss","method_name":"Huber loss"},{"method_slug":"iql","method_name":"IQL"},{"method_slug":"q-learning","method_name":"Q-Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.12955","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.12955"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/YangRui2015/RIQL","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zzmtsvv/ORL","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yangrui2015/riql","reach":null}],"summary":{"ran_draft_wrong":2,"ran":2,"ran_honours":1,"unverified":4},"by_repo_kind":{"official":{"samples":5,"ran":2,"repositories":1},"listed":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"639b4986c270df90","entry":"compute_mean_std","repo":"YangRui2015/RIQL","repo_kind":"official","path":"RIQL.py","file_url":"https://github.com/YangRui2015/RIQL/blob/HEAD/RIQL.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"639b4986c270df90"}},{"code_sha256_prefix":"3a5956686dfd460b","entry":"Actor","repo":"zzmtsvv/ORL","repo_kind":"listed","path":"riql/riql.py","file_url":"https://github.com/zzmtsvv/ORL/blob/HEAD/riql/riql.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3a5956686dfd460b"}},{"code_sha256_prefix":"b7c0c6c1a8acecf2","entry":"asdict","repo":"YangRui2015/RIQL","repo_kind":"official","path":"RIQL.py","file_url":"https://github.com/YangRui2015/RIQL/blob/HEAD/RIQL.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b7c0c6c1a8acecf2"}},{"code_sha256_prefix":"ba80f9ff81b1de5f","entry":"normalize_states","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"ba80f9ff81b1de5f"}},{"code_sha256_prefix":"1a1679fcb96b8d82","entry":"riql_config","repo":"zzmtsvv/ORL","repo_kind":"listed","path":"riql/riql.py","file_url":"https://github.com/zzmtsvv/ORL/blob/HEAD/riql/riql.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1a1679fcb96b8d82"}},{"code_sha256_prefix":"25ff9107d049cbeb","entry":"ImplicitQLearning","repo":"yangrui2015/riql","repo_kind":"official","path":"RIQL.py","file_url":"https://github.com/yangrui2015/riql/blob/HEAD/RIQL.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"25ff9107d049cbeb"}},{"code_sha256_prefix":"835272ed168c5903","entry":"RIQL","repo":"zzmtsvv/ORL","repo_kind":"listed","path":"riql/riql.py","file_url":"https://github.com/zzmtsvv/ORL/blob/HEAD/riql/riql.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"835272ed168c5903"}},{"code_sha256_prefix":"121a70492e315410","entry":"TrainConfig","repo":"yangrui2015/riql","repo_kind":"official","path":"RIQL.py","file_url":"https://github.com/yangrui2015/riql/blob/HEAD/RIQL.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"121a70492e315410"}},{"code_sha256_prefix":"cca4505f10b19283","entry":"soft_update","repo":"yangrui2015/riql","repo_kind":"official","path":"RIQL.py","file_url":"https://github.com/yangrui2015/riql/blob/HEAD/RIQL.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cca4505f10b19283"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}