{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/off-policy-deep-reinforcement-learning","title":"Off-Policy Deep Reinforcement Learning without Exploration","arxiv_id":"1812.02900","date":"2018-12-07","proceeding":null,"authors":["Scott Fujimoto","David Meger","Doina Precup"],"abstract":"Many practical applications of reinforcement learning constrain agents to learn from a fixed batch of data which has already been gathered, without offering further possibility for data collection. In this paper, we demonstrate that due to errors introduced by extrapolation, standard off-policy deep reinforcement learning algorithms, such as DQN and DDPG, are incapable of learning with data uncorrelated to the distribution under the current policy, making them ineffective for this fixed batch setting. We introduce a novel class of off-policy algorithms, batch-constrained reinforcement learning, which restricts the action space in order to force the agent towards behaving close to on-policy with respect to a subset of the given data. We present the first continuous control deep reinforcement learning algorithm which can learn effectively from arbitrary, fixed batch data, and empirically demonstrate the quality of its behavior in several tasks.","url_abs":"https://arxiv.org/abs/1812.02900v3","url_pdf":"https://arxiv.org/pdf/1812.02900v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"off-policy-deep-reinforcement-learning","repo_url":"https://github.com/sfujim/BCQ","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"off-policy-deep-reinforcement-learning","repo_url":"https://github.com/AurelianTactics/bcq_tensorflow","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"off-policy-deep-reinforcement-learning","repo_url":"https://github.com/haosulab/ManiSkill-Learn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"off-policy-deep-reinforcement-learning","repo_url":"https://github.com/massquantity/DBRL","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"off-policy-deep-reinforcement-learning","repo_url":"https://github.com/maziarg/PrivAttack-BCQ","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"off-policy-deep-reinforcement-learning","repo_url":"https://github.com/sparkmxy/my-offlinerl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"off-policy-deep-reinforcement-learning","repo_url":"https://github.com/takuseno/d3rlpy","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"off-policy-deep-reinforcement-learning","repo_url":"https://github.com/theSparta/off_policy_mujoco","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"off-policy-deep-reinforcement-learning","repo_url":"https://github.com/thxsxth/POMDP_RLSepsis","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"off-policy-deep-reinforcement-learning","repo_url":"https://github.com/HzcIrving/DLRL-PlayGround/tree/main/Offline%20RL/BCQ","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"continuous-control","task_name":"continuous-control"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"ddpg","method_name":"DDPG"},{"method_slug":"dqn","method_name":"DQN"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"experience-replay","method_name":"Experience Replay"},{"method_slug":"q-learning","method_name":"Q-Learning"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1812.02900","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1812.02900"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/takuseno/d3rlpy","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/sfujim/BCQ","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/HzcIrving/DLRL-PlayGround/tree/main/Offline%20RL/BCQ","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/maziarg/PrivAttack-BCQ","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sparkmxy/my-offlinerl","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/theSparta/off_policy_mujoco","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AurelianTactics/bcq_tensorflow","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/haosulab/ManiSkill-Learn","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/massquantity/DBRL","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thxsxth/POMDP_RLSepsis","reach":null}],"summary":{"ran":12,"ran_honours":1,"unverified":1},"by_repo_kind":{"listed":{"samples":14,"ran":13,"repositories":6}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"a4c895a75dcac04b","entry":"Actor","repo":"theSparta/off_policy_mujoco","repo_kind":"listed","path":"RSEM.py","file_url":"https://github.com/theSparta/off_policy_mujoco/blob/HEAD/RSEM.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a4c895a75dcac04b"}},{"code_sha256_prefix":"fa0f609acc14b1b7","entry":"Actor","repo":"maziarg/PrivAttack-BCQ","repo_kind":"listed","path":"continuous_BCQ/BCQ.py","file_url":"https://github.com/maziarg/PrivAttack-BCQ/blob/HEAD/continuous_BCQ/BCQ.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fa0f609acc14b1b7"}},{"code_sha256_prefix":"f8f318e0c81d2112","entry":"Actor","repo":"thxsxth/POMDP_RLSepsis","repo_kind":"listed","path":"RL/other RL attempts/BCQ_models.py","file_url":"https://github.com/thxsxth/POMDP_RLSepsis/blob/HEAD/RL/other%20RL%20attempts/BCQ_models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f8f318e0c81d2112"}},{"code_sha256_prefix":"c9c4a9b681f93589","entry":"BCQ","repo":"massquantity/DBRL","repo_kind":"listed","path":"dbrl/models/bcq.py","file_url":"https://github.com/massquantity/DBRL/blob/HEAD/dbrl/models/bcq.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c9c4a9b681f93589"}},{"code_sha256_prefix":"b1e9a6c042a3aa75","entry":"BCQ","repo":"maziarg/PrivAttack-BCQ","repo_kind":"listed","path":"continuous_BCQ/BCQ.py","file_url":"https://github.com/maziarg/PrivAttack-BCQ/blob/HEAD/continuous_BCQ/BCQ.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1e9a6c042a3aa75"}},{"code_sha256_prefix":"c582768c3f585aac","entry":"BCQ","repo":"thxsxth/POMDP_RLSepsis","repo_kind":"listed","path":"RL/other RL attempts/BCQ_models.py","file_url":"https://github.com/thxsxth/POMDP_RLSepsis/blob/HEAD/RL/other%20RL%20attempts/BCQ_models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c582768c3f585aac"}},{"code_sha256_prefix":"d0ff87764918534f","entry":"BCQPolicy","repo":"sparkmxy/my-offlinerl","repo_kind":"listed","path":"offlinerl/algo/modelfree/bcq.py","file_url":"https://github.com/sparkmxy/my-offlinerl/blob/HEAD/offlinerl/algo/modelfree/bcq.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d0ff87764918534f"}},{"code_sha256_prefix":"5bd57f3d9b802889","entry":"Critic","repo":"theSparta/off_policy_mujoco","repo_kind":"listed","path":"RSEM.py","file_url":"https://github.com/theSparta/off_policy_mujoco/blob/HEAD/RSEM.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5bd57f3d9b802889"}},{"code_sha256_prefix":"48b14528ccc28930","entry":"Critic","repo":"maziarg/PrivAttack-BCQ","repo_kind":"listed","path":"continuous_BCQ/BCQ.py","file_url":"https://github.com/maziarg/PrivAttack-BCQ/blob/HEAD/continuous_BCQ/BCQ.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"48b14528ccc28930"}},{"code_sha256_prefix":"39b2df9b5836adaa","entry":"RSEM","repo":"theSparta/off_policy_mujoco","repo_kind":"listed","path":"RSEM.py","file_url":"https://github.com/theSparta/off_policy_mujoco/blob/HEAD/RSEM.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39b2df9b5836adaa"}},{"code_sha256_prefix":"a10b65bd9d484e03","entry":"VAE","repo":"maziarg/PrivAttack-BCQ","repo_kind":"listed","path":"continuous_BCQ/BCQ.py","file_url":"https://github.com/maziarg/PrivAttack-BCQ/blob/HEAD/continuous_BCQ/BCQ.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a10b65bd9d484e03"}},{"code_sha256_prefix":"e3a9f4f45dc4af4e","entry":"VAE","repo":"thxsxth/POMDP_RLSepsis","repo_kind":"listed","path":"RL/other RL attempts/BCQ_models.py","file_url":"https://github.com/thxsxth/POMDP_RLSepsis/blob/HEAD/RL/other%20RL%20attempts/BCQ_models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e3a9f4f45dc4af4e"}},{"code_sha256_prefix":"b5629ba35995f571","entry":"simplex_sample","repo":"theSparta/off_policy_mujoco","repo_kind":"listed","path":"RSEM.py","file_url":"https://github.com/theSparta/off_policy_mujoco/blob/HEAD/RSEM.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b5629ba35995f571"}},{"code_sha256_prefix":"8c05bd3cd12f39cd","entry":"BCQ","repo":"HzcIrving/DLRL-PlayGround","repo_kind":"listed","path":"Offline RL/BCQ/BCQ.py","file_url":"https://github.com/HzcIrving/DLRL-PlayGround/blob/HEAD/Offline%20RL/BCQ/BCQ.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c05bd3cd12f39cd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}