{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/offline-rl-with-no-ood-actions-in-sample","title":"Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization","arxiv_id":"2303.15810","date":"2023-03-28","proceeding":null,"authors":["Haoran Xu","Li Jiang","Jianxiong Li","Zhuoran Yang","Zhaoran Wang","Victor Wai Kin Chan","Xianyuan Zhan"],"abstract":"Most offline reinforcement learning (RL) methods suffer from the trade-off between improving the policy to surpass the behavior policy and constraining the policy to limit the deviation from the behavior policy as computing $Q$-values using out-of-distribution (OOD) actions will suffer from errors due to distributional shift. The recently proposed \\textit{In-sample Learning} paradigm (i.e., IQL), which improves the policy by quantile regression using only data samples, shows great promise because it learns an optimal policy without querying the value function of any unseen actions. However, it remains unclear how this type of method handles the distributional shift in learning the value function. In this work, we make a key finding that the in-sample learning paradigm arises under the \\textit{Implicit Value Regularization} (IVR) framework. This gives a deeper understanding of why the in-sample learning paradigm works, i.e., it applies implicit value regularization to the policy. Based on the IVR framework, we further propose two practical algorithms, Sparse $Q$-learning (SQL) and Exponential $Q$-learning (EQL), which adopt the same value regularization used in existing works, but in a complete in-sample manner. Compared with IQL, we find that our algorithms introduce sparsity in learning the value function, making them more robust in noisy data regimes. We also verify the effectiveness of SQL and EQL on D4RL benchmark datasets and show the benefits of in-sample learning by comparing them with CQL in small data regimes.","url_abs":"https://arxiv.org/abs/2303.15810v1","url_pdf":"https://arxiv.org/pdf/2303.15810v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"offline-rl-with-no-ood-actions-in-sample","repo_url":"https://github.com/ryanxhr/ivr","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"offline-rl-with-no-ood-actions-in-sample","repo_url":"https://github.com/linhlpv/Offline-and-OfflineOnline-RL","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"offline-rl-with-no-ood-actions-in-sample","repo_url":"https://github.com/perrin-isir/afu","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"offline-rl-with-no-ood-actions-in-sample","repo_url":"https://github.com/zzmtsvv/rl_task","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"d4rl","task_name":"D4RL"},{"task_slug":"offline-rl","task_name":"Offline RL"},{"task_slug":"q-learning","task_name":"Q-Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"quantile-regression","task_name":"quantile regression"}],"methods":[{"method_slug":"iql","method_name":"IQL"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2303.15810","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.15810"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ryanxhr/IVR","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/perrin-isir/afu","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/linhlpv/Offline-and-OfflineOnline-RL","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zzmtsvv/rl_task","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ryanxhr/ivr","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"718462716d78ea99","entry":"default_init","repo":"ryanxhr/IVR","repo_kind":"official","path":"common.py","file_url":"https://github.com/ryanxhr/IVR/blob/HEAD/common.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"718462716d78ea99"}},{"code_sha256_prefix":"0c3b9f06d1f100cd","entry":"merge_trajectories","repo":"ryanxhr/IVR","repo_kind":"official","path":"dataset_utils.py","file_url":"https://github.com/ryanxhr/IVR/blob/HEAD/dataset_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c3b9f06d1f100cd"}},{"code_sha256_prefix":"b5cd4f789150335c","entry":"split_into_trajectories","repo":"ryanxhr/IVR","repo_kind":"official","path":"dataset_utils.py","file_url":"https://github.com/ryanxhr/IVR/blob/HEAD/dataset_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b5cd4f789150335c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}