{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/imitation-from-arbitrary-experience-a-dual","title":"Dual RL: Unification and New Methods for Reinforcement and Imitation Learning","arxiv_id":"2302.08560","date":"2023-02-16","proceeding":null,"authors":["Harshit Sikchi","Qinqing Zheng","Amy Zhang","Scott Niekum"],"abstract":"The goal of reinforcement learning (RL) is to find a policy that maximizes the expected cumulative return. It has been shown that this objective can be represented as an optimization problem of state-action visitation distribution under linear constraints. The dual problem of this formulation, which we refer to as dual RL, is unconstrained and easier to optimize. In this work, we first cast several state-of-the-art offline RL and offline imitation learning (IL) algorithms as instances of dual RL approaches with shared structures. Such unification allows us to identify the root cause of the shortcomings of prior methods. For offline IL, our analysis shows that prior methods are based on a restrictive coverage assumption that greatly limits their performance in practice. To fix this limitation, we propose a new discriminator-free method ReCOIL that learns to imitate from arbitrary off-policy data to obtain near-expert performance. For offline RL, our analysis frames a recent offline RL method XQL in the dual framework, and we further propose a new method f-DVL that provides alternative choices to the Gumbel regression loss that fixes the known training instability issue of XQL. The performance improvements by both of our proposed methods, ReCOIL and f-DVL, in IL and RL are validated on an extensive suite of simulated robot locomotion and manipulation tasks. Project code and details can be found at this https://hari-sikchi.github.io/dual-rl.","url_abs":"https://arxiv.org/abs/2302.08560v3","url_pdf":"https://arxiv.org/pdf/2302.08560v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"imitation-from-arbitrary-experience-a-dual","repo_url":"https://github.com/hari-sikchi/DVL","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null}],"tasks":[{"task_slug":"imitation-learning","task_name":"Imitation Learning"},{"task_slug":"offline-rl","task_name":"Offline RL"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2302.08560","atlas_url":"https://app.syntology.ai/?focus=2302.08560","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2302.08560"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hari-sikchi/DVL","reach":null}],"summary":{"ran_violates":2,"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"143f39ac7129288e","entry":"chi_square_loss","repo":"hari-sikchi/DVL","repo_kind":"official","path":"offline/critic.py","file_url":"https://github.com/hari-sikchi/DVL/blob/HEAD/offline/critic.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"143f39ac7129288e"}},{"code_sha256_prefix":"53d821721fbb5fe1","entry":"reverse_kl_loss","repo":"hari-sikchi/DVL","repo_kind":"official","path":"offline/critic.py","file_url":"https://github.com/hari-sikchi/DVL/blob/HEAD/offline/critic.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"53d821721fbb5fe1"}},{"code_sha256_prefix":"d773884e980e01c5","entry":"total_variation_loss","repo":"hari-sikchi/DVL","repo_kind":"official","path":"offline/critic.py","file_url":"https://github.com/hari-sikchi/DVL/blob/HEAD/offline/critic.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d773884e980e01c5"}},{"code_sha256_prefix":"d3aa5cd1dcb8eff9","entry":"update_v","repo":"hari-sikchi/DVL","repo_kind":"official","path":"offline/critic.py","file_url":"https://github.com/hari-sikchi/DVL/blob/HEAD/offline/critic.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d3aa5cd1dcb8eff9"}},{"code_sha256_prefix":"ac1979a869131b2d","entry":"Model","repo":"hari-sikchi/DVL","repo_kind":"official","path":"offline/critic.py","file_url":"https://github.com/hari-sikchi/DVL/blob/HEAD/offline/critic.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ac1979a869131b2d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}