{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hybrid-rl-using-both-offline-and-online-data","title":"Hybrid RL: Using Both Offline and Online Data Can Make RL Efficient","arxiv_id":"2210.06718","date":"2022-10-13","proceeding":null,"authors":["Yuda Song","Yifei Zhou","Ayush Sekhari","J. Andrew Bagnell","Akshay Krishnamurthy","Wen Sun"],"abstract":"We consider a hybrid reinforcement learning setting (Hybrid RL), in which an agent has access to an offline dataset and the ability to collect experience via real-world online interaction. The framework mitigates the challenges that arise in both pure offline and online RL settings, allowing for the design of simple and highly effective algorithms, in both theory and practice. We demonstrate these advantages by adapting the classical Q learning/iteration algorithm to the hybrid setting, which we call Hybrid Q-Learning or Hy-Q. In our theoretical results, we prove that the algorithm is both computationally and statistically efficient whenever the offline dataset supports a high-quality policy and the environment has bounded bilinear rank. Notably, we require no assumptions on the coverage provided by the initial distribution, in contrast with guarantees for policy gradient/iteration methods. In our experimental results, we show that Hy-Q with neural network function approximation outperforms state-of-the-art online, offline, and hybrid RL baselines on challenging benchmarks, including Montezuma's Revenge.","url_abs":"https://arxiv.org/abs/2210.06718v3","url_pdf":"https://arxiv.org/pdf/2210.06718v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hybrid-rl-using-both-offline-and-online-data","repo_url":"https://github.com/yudasong/hyq","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"montezumas-revenge","task_name":"Montezuma's Revenge"},{"task_slug":"q-learning","task_name":"Q-Learning"}],"methods":[{"method_slug":"q-learning","method_name":"Q-Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2210.06718","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.06718"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/yudasong/HyQ","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/BY571/CQL","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/yudasong/briee","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yudasong/hyq","reach":{"status":"ok"}}],"summary":{"ran_honours":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"4f301a6582d71f3d","entry":"eval","repo":"yudasong/HyQ","repo_kind":"official","path":"MRevenge/train_atari.py","file_url":"https://github.com/yudasong/HyQ/blob/HEAD/MRevenge/train_atari.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4f301a6582d71f3d"}},{"code_sha256_prefix":"ea623cc39c0fd92d","entry":"evaluate","repo":"yudasong/HyQ","repo_kind":"official","path":"CLock/online_main_lock.py","file_url":"https://github.com/yudasong/HyQ/blob/HEAD/CLock/online_main_lock.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ea623cc39c0fd92d"}},{"code_sha256_prefix":"fccb84a1f13f02c3","entry":"fqi_update","repo":"yudasong/HyQ","repo_kind":"official","path":"CLock/online_main_lock.py","file_url":"https://github.com/yudasong/HyQ/blob/HEAD/CLock/online_main_lock.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fccb84a1f13f02c3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}