{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/190600949","title":"Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction","arxiv_id":"1906.00949","date":"2019-06-03","proceeding":"NeurIPS 2019 12","authors":["Aviral Kumar","Justin Fu","George Tucker","Sergey Levine"],"abstract":"Off-policy reinforcement learning aims to leverage experience collected from prior policies for sample-efficient learning. However, in practice, commonly used off-policy approximate dynamic programming methods based on Q-learning and actor-critic methods are highly sensitive to the data distribution, and can make only limited progress without collecting additional on-policy data. As a step towards more robust off-policy algorithms, we study the setting where the off-policy experience is fixed and there is no further interaction with the environment. We identify bootstrapping error as a key source of instability in current methods. Bootstrapping error is due to bootstrapping from actions that lie outside of the training data distribution, and it accumulates via the Bellman backup operator. We theoretically analyze bootstrapping error, and demonstrate how carefully constraining action selection in the backup can mitigate it. Based on our analysis, we propose a practical algorithm, bootstrapping error accumulation reduction (BEAR). We demonstrate that BEAR is able to learn robustly from different off-policy distributions, including random and suboptimal demonstrations, on a range of continuous control tasks.","url_abs":"https://arxiv.org/abs/1906.00949v2","url_pdf":"https://arxiv.org/pdf/1906.00949v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"190600949","repo_url":"https://github.com/aviralkumar2907/BEAR","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"190600949","repo_url":"https://github.com/takuseno/d3rlpy","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"190600949","repo_url":"https://github.com/zzmtsvv/rl_task","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"q-learning","task_name":"Q-Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"continuous-control","task_name":"continuous-control"}],"methods":[{"method_slug":"q-learning","method_name":"Q-Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1906.00949","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1906.00949"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/takuseno/d3rlpy","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aviralkumar2907/BEAR","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zzmtsvv/rl_task","reach":null}],"summary":{"ran":4,"unverified":1},"by_repo_kind":{"listed":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"434fd896deb0893a","entry":"Actor","repo":"zzmtsvv/rl_task","repo_kind":"listed","path":"bear/bear.py","file_url":"https://github.com/zzmtsvv/rl_task/blob/HEAD/bear/bear.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"434fd896deb0893a"}},{"code_sha256_prefix":"b68b870205409436","entry":"EnsembledCritic","repo":"zzmtsvv/rl_task","repo_kind":"listed","path":"bear/bear.py","file_url":"https://github.com/zzmtsvv/rl_task/blob/HEAD/bear/bear.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b68b870205409436"}},{"code_sha256_prefix":"8e8aaff4b899322e","entry":"EnsembledLinear","repo":"zzmtsvv/rl_task","repo_kind":"listed","path":"bear/bear.py","file_url":"https://github.com/zzmtsvv/rl_task/blob/HEAD/bear/bear.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8e8aaff4b899322e"}},{"code_sha256_prefix":"1864c674e3689f9e","entry":"bear_config","repo":"zzmtsvv/rl_task","repo_kind":"listed","path":"bear/bear.py","file_url":"https://github.com/zzmtsvv/rl_task/blob/HEAD/bear/bear.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1864c674e3689f9e"}},{"code_sha256_prefix":"7d071fb8cb89ee70","entry":"BEAR","repo":"zzmtsvv/rl_task","repo_kind":"listed","path":"bear/bear.py","file_url":"https://github.com/zzmtsvv/rl_task/blob/HEAD/bear/bear.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7d071fb8cb89ee70"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}