{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dr-jekyll-and-mr-hyde-the-strange-case-of-off","title":"Dr Jekyll and Mr Hyde: the Strange Case of Off-Policy Policy Updates","arxiv_id":"2109.14727","date":"2021-09-29","proceeding":null,"authors":["Romain Laroche","Remi Tachet"],"abstract":"The policy gradient theorem states that the policy should only be updated in states that are visited by the current policy, which leads to insufficient planning in the off-policy states, and thus to convergence to suboptimal policies. We tackle this planning issue by extending the policy gradient theory to policy updates with respect to any state density. Under these generalized policy updates, we show convergence to optimality under a necessary and sufficient condition on the updates' state densities, and thereby solve the aforementioned planning issue. We also prove asymptotic convergence rates that significantly improve those in the policy gradient literature. To implement the principles prescribed by our theory, we propose an agent, Dr Jekyll & Mr Hyde (JH), with a double personality: Dr Jekyll purely exploits while Mr Hyde purely explores. JH's independent policies allow to record two separate replay buffers: one on-policy (Dr Jekyll's) and one off-policy (Mr Hyde's), and therefore to update JH's models with a mixture of on-policy and off-policy updates. More than an algorithm, JH defines principles for actor-critic algorithms to satisfy the requirements we identify in our analysis. We extensively test on finite MDPs where JH demonstrates a superior ability to recover from converging to a suboptimal policy without impairing its speed of convergence. We also implement a deep version of the algorithm and test it on a simple problem where it shows promising results.","url_abs":"https://arxiv.org/abs/2109.14727v1","url_pdf":"https://arxiv.org/pdf/2109.14727v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dr-jekyll-and-mr-hyde-the-strange-case-of-off","repo_url":"https://github.com/microsoft/dr-jekyll-and-mr-hyde-the-strange-case-of-off-policy-policy-updates","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2109.14727","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2109.14727"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/dr-jekyll-and-mr-hyde-the-strange-case-of-off-policy-policy-updates","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"368ab0000c0fc37c","entry":"choice","repo":"microsoft/dr-jekyll-and-mr-hyde-the-strange-case-of-off-policy-policy-updates","repo_kind":"official","path":"FiniteJH/utils.py","file_url":"https://github.com/microsoft/dr-jekyll-and-mr-hyde-the-strange-case-of-off-policy-policy-updates/blob/HEAD/FiniteJH/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"368ab0000c0fc37c"}},{"code_sha256_prefix":"74d7448247398145","entry":"get_reward_model","repo":"microsoft/dr-jekyll-and-mr-hyde-the-strange-case-of-off-policy-policy-updates","repo_kind":"official","path":"FiniteJH/utils.py","file_url":"https://github.com/microsoft/dr-jekyll-and-mr-hyde-the-strange-case-of-off-policy-policy-updates/blob/HEAD/FiniteJH/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"74d7448247398145"}},{"code_sha256_prefix":"9489b9875fccd8f1","entry":"projection_simplex_sort","repo":"microsoft/dr-jekyll-and-mr-hyde-the-strange-case-of-off-policy-policy-updates","repo_kind":"official","path":"FiniteJH/proj_simplex.py","file_url":"https://github.com/microsoft/dr-jekyll-and-mr-hyde-the-strange-case-of-off-policy-policy-updates/blob/HEAD/FiniteJH/proj_simplex.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9489b9875fccd8f1"}},{"code_sha256_prefix":"6f9d3eeb12cb5742","entry":"softmax_action","repo":"microsoft/dr-jekyll-and-mr-hyde-the-strange-case-of-off-policy-policy-updates","repo_kind":"official","path":"FiniteJH/utils.py","file_url":"https://github.com/microsoft/dr-jekyll-and-mr-hyde-the-strange-case-of-off-policy-policy-updates/blob/HEAD/FiniteJH/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6f9d3eeb12cb5742"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}