{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/coptidice-offline-constrained-reinforcement-1","title":"COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimation","arxiv_id":"2204.08957","date":"2022-04-19","proceeding":"ICLR 2022 4","authors":["Jongmin Lee","Cosmin Paduraru","Daniel J. Mankowitz","Nicolas Heess","Doina Precup","Kee-Eung Kim","Arthur Guez"],"abstract":"We consider the offline constrained reinforcement learning (RL) problem, in which the agent aims to compute a policy that maximizes expected return while satisfying given cost constraints, learning only from a pre-collected dataset. This problem setting is appealing in many real-world scenarios, where direct interaction with the environment is costly or risky, and where the resulting policy should comply with safety constraints. However, it is challenging to compute a policy that guarantees satisfying the cost constraints in the offline RL setting, since the off-policy evaluation inherently has an estimation error. In this paper, we present an offline constrained RL algorithm that optimizes the policy in the space of the stationary distribution. Our algorithm, COptiDICE, directly estimates the stationary distribution corrections of the optimal policy with respect to returns, while constraining the cost upper bound, with the goal of yielding a cost-conservative policy for actual constraint satisfaction. Experimental results show that COptiDICE attains better policies in terms of constraint satisfaction and return-maximization, outperforming baseline algorithms.","url_abs":"https://arxiv.org/abs/2204.08957v1","url_pdf":"https://arxiv.org/pdf/2204.08957v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"coptidice-offline-constrained-reinforcement-1","repo_url":"https://github.com/deepmind/constrained_optidice","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"off-policy-evaluation","task_name":"Off-policy evaluation"},{"task_slug":"offline-rl","task_name":"Offline RL"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2204.08957","atlas_url":"https://app.syntology.ai/?focus=2204.08957","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2204.08957"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/deepmind/constrained_optidice","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4,"unverified":3},"by_repo_kind":{"official":{"samples":7,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"021d008f035670a2","entry":"domain_cost_fn","repo":"deepmind/constrained_optidice","repo_kind":"official","path":"neural/cost.py","file_url":"https://github.com/deepmind/constrained_optidice/blob/HEAD/neural/cost.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"021d008f035670a2"}},{"code_sha256_prefix":"b0a316d25b4cad24","entry":"get_f_divergence_fn","repo":"deepmind/constrained_optidice","repo_kind":"official","path":"neural/learning.py","file_url":"https://github.com/deepmind/constrained_optidice/blob/HEAD/neural/learning.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b0a316d25b4cad24"}},{"code_sha256_prefix":"77a7c8ddd9479170","entry":"policy_evaluation","repo":"deepmind/constrained_optidice","repo_kind":"official","path":"tabular/mdp_util.py","file_url":"https://github.com/deepmind/constrained_optidice/blob/HEAD/tabular/mdp_util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"77a7c8ddd9479170"}},{"code_sha256_prefix":"3bb97e9b8f8bef6d","entry":"policy_evaluation_mdp","repo":"deepmind/constrained_optidice","repo_kind":"official","path":"tabular/mdp_util.py","file_url":"https://github.com/deepmind/constrained_optidice/blob/HEAD/tabular/mdp_util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3bb97e9b8f8bef6d"}},{"code_sha256_prefix":"68bce47fe9bdd627","entry":"conditional_update","repo":"deepmind/constrained_optidice","repo_kind":"official","path":"neural/learning.py","file_url":"https://github.com/deepmind/constrained_optidice/blob/HEAD/neural/learning.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"68bce47fe9bdd627"}},{"code_sha256_prefix":"06a80085f44b4edd","entry":"generate_random_cmdp","repo":"deepmind/constrained_optidice","repo_kind":"official","path":"tabular/mdp_util.py","file_url":"https://github.com/deepmind/constrained_optidice/blob/HEAD/tabular/mdp_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"06a80085f44b4edd"}},{"code_sha256_prefix":"ba29d1b163a7d24d","entry":"periodic_update","repo":"deepmind/constrained_optidice","repo_kind":"official","path":"neural/learning.py","file_url":"https://github.com/deepmind/constrained_optidice/blob/HEAD/neural/learning.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ba29d1b163a7d24d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}