{"url":"/method/double-q-learning","slug":"double-q-learning","name":"Double Q-learning","full_name":"Double Q-learning","full_name_withheld":false,"description_markdown":"**Double Q-learning** is an off-policy reinforcement learning algorithm that utilises double estimation to counteract overestimation problems with traditional Q-learning. \r\n\r\nThe max operator in standard [Q-learning](https://paperswithcode.com/method/q-learning) and [DQN](https://paperswithcode.com/method/dqn) uses the same values both to select and to evaluate an action. This makes it more likely to select overestimated values, resulting in overoptimistic value estimates. To prevent this, we can decouple the selection from the evaluation, which is the idea behind Double Q-learning:\r\n\r\n$$ Y^{Q}\\_{t} = R\\_{t+1} + \\gamma{Q}\\left(S\\_{t+1}, \\arg\\max\\_{a}Q\\left(S\\_{t+1}, a; \\mathbb{\\theta}\\_{t}\\right);\\mathbb{\\theta}\\_{t}\\right) $$\r\n\r\nThe Double Q-learning error can then be written as:\r\n\r\n$$ Y^{DoubleQ}\\_{t} = R\\_{t+1} + \\gamma{Q}\\left(S\\_{t+1}, \\arg\\max\\_{a}Q\\left(S\\_{t+1}, a; \\mathbb{\\theta}\\_{t}\\right);\\mathbb{\\theta}^{'}\\_{t}\\right) $$\r\n\r\nHere the selection of the action in the $\\arg\\max$ is still due to the online weights $\\theta\\_{t}$. But we use a second set of weights $\\mathbb{\\theta}^{'}\\_{t}$ to fairly evaluate the value of this policy.\r\n\r\nSource: [Deep Reinforcement Learning with Double Q-learning](https://paperswithcode.com/paper/deep-reinforcement-learning-with-double-q)","description_state":"present","introduced_year":null,"introduced_by":{"title":"Double Q-learning","paper":"/paper/double-q-learning","first_author":"Hado V. Hasselt","n_authors":1,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/double-q-learning"},"source":{"url":"http://papers.nips.cc/paper/3964-double-q-learning","title":"Double Q-learning","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Reinforcement Learning","area_id":"reinforcement-learning","collection":"Off-Policy TD Control","url":"/methods/category/off-policy-td-control","pwc_aliases":[]}],"n_papers_tagged":112,"archive_num_papers":112,"papers_newest_first":[{"paper":null,"title":"Reinforcement Learning-Based Policy Optimisation For Heterogeneous Radio Access","date":"2025-06-18","arxiv_id":"2506.15273","n_code_links":0,"syntology":null},{"paper":null,"title":"Dynamic Operating System Scheduling Using Double DQN: A Reinforcement Learning Approach to Task Optimization","date":"2025-03-31","arxiv_id":"2503.23659","n_code_links":0,"syntology":null},{"paper":null,"title":"Distribution-Free Uncertainty Quantification in Mechanical Ventilation Treatment: A Conformal Deep Q-Learning Framework","date":"2024-12-17","arxiv_id":"2412.12597","n_code_links":0,"syntology":null},{"paper":"/paper/beyond-the-rainbow-high-performance-deep","title":"Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC","date":"2024-11-06","arxiv_id":"2411.03820","n_code_links":3,"syntology":{"ran":13,"of":25,"unverified":12,"pointer_only":21}},{"paper":null,"title":"Bootstrapping Expectiles in Reinforcement Learning","date":"2024-06-06","arxiv_id":"2406.04081","n_code_links":0,"syntology":null},{"paper":null,"title":"A New View on Planning in Online Reinforcement Learning","date":"2024-06-03","arxiv_id":"2406.01562","n_code_links":0,"syntology":null},{"paper":null,"title":"Active search and coverage using point-cloud reinforcement learning","date":"2023-12-18","arxiv_id":"2312.11410","n_code_links":0,"syntology":null},{"paper":"/paper/efficient-sparse-reward-goal-conditioned","title":"Efficient Sparse-Reward Goal-Conditioned Reinforcement Learning with a High Replay Ratio and Regularization","date":"2023-12-10","arxiv_id":"2312.05787","n_code_links":1,"syntology":null},{"paper":null,"title":"Data-efficient Deep Reinforcement Learning for Vehicle Trajectory Control","date":"2023-11-30","arxiv_id":"2311.18393","n_code_links":0,"syntology":null},{"paper":null,"title":"Advancing Algorithmic Trading: A Multi-Technique Enhancement of Deep Q-Network Models","date":"2023-11-09","arxiv_id":"2311.05743","n_code_links":0,"syntology":null},{"paper":null,"title":"Deep Reinforcement Learning for the Heat Transfer Control of Pulsating Impinging Jets","date":"2023-09-25","arxiv_id":"2309.13955","n_code_links":0,"syntology":null},{"paper":null,"title":"Adaptive Multi-Agent Deep Reinforcement Learning for Timely Healthcare Interventions","date":"2023-09-20","arxiv_id":"2309.10980","n_code_links":0,"syntology":null},{"paper":null,"title":"Deep Reinforcement Learning for Artificial Upwelling Energy Management","date":"2023-08-20","arxiv_id":"2308.10199","n_code_links":0,"syntology":null},{"paper":null,"title":"Interpretable and Secure Trajectory Optimization for UAV-Assisted Communication","date":"2023-07-05","arxiv_id":"2307.02002","n_code_links":0,"syntology":null},{"paper":null,"title":"Optimizing Credit Limit Adjustments Under Adversarial Goals Using Reinforcement Learning","date":"2023-06-27","arxiv_id":"2306.15585","n_code_links":0,"syntology":null},{"paper":null,"title":"Vanishing Bias Heuristic-guided Reinforcement Learning Algorithm","date":"2023-06-17","arxiv_id":"2306.10216","n_code_links":0,"syntology":null},{"paper":null,"title":"RSRM: Reinforcement Symbolic Regression Machine","date":"2023-05-24","arxiv_id":"2305.14656","n_code_links":0,"syntology":null},{"paper":"/paper/extracting-diagnosis-pathways-from-electronic","title":"Extracting Diagnosis Pathways from Electronic Health Records Using Deep Reinforcement Learning","date":"2023-05-10","arxiv_id":"2305.06295","n_code_links":1,"syntology":null},{"paper":"/paper/train-a-real-world-local-path-planner-in-one","title":"Train a Real-world Local Path Planner in One Hour via Partially Decoupled Reinforcement Learning and Vectorized Diversity","date":"2023-05-07","arxiv_id":"2305.04180","n_code_links":1,"syntology":null},{"paper":null,"title":"Smoothed Q-learning","date":"2023-03-15","arxiv_id":"2303.08631","n_code_links":0,"syntology":null},{"paper":null,"title":"Double A3C: Deep Reinforcement Learning on OpenAI Gym Games","date":"2023-03-04","arxiv_id":"2303.02271","n_code_links":0,"syntology":null},{"paper":null,"title":"Asynchronous Deep Double Duelling Q-Learning for Trading-Signal Execution in Limit Order Book Markets","date":"2023-01-20","arxiv_id":"2301.08688","n_code_links":0,"syntology":null},{"paper":null,"title":"Hierarchical Deep Q-Learning Based Handover in Wireless Networks with Dual Connectivity","date":"2023-01-13","arxiv_id":"2301.05391","n_code_links":0,"syntology":null},{"paper":null,"title":"Neighboring state-based RL Exploration","date":"2022-12-21","arxiv_id":"2212.10712","n_code_links":0,"syntology":null},{"paper":null,"title":"Elastic Step DQN: A novel multi-step algorithm to alleviate overestimation in Deep QNetworks","date":"2022-10-07","arxiv_id":"2210.03325","n_code_links":0,"syntology":null},{"paper":"/paper/m-2-dqn-a-robust-method-for-accelerating-deep","title":"M$^2$DQN: A Robust Method for Accelerating Deep Q-learning Network","date":"2022-09-16","arxiv_id":"2209.07809","n_code_links":1,"syntology":null},{"paper":"/paper/reducing-variance-in-temporal-difference","title":"Reducing Variance in Temporal-Difference Value Estimation via Ensemble of Deep Networks","date":"2022-09-16","arxiv_id":"2209.07670","n_code_links":1,"syntology":null},{"paper":null,"title":"Double Q-Learning for Citizen Relocation During Natural Hazards","date":"2022-09-08","arxiv_id":"2209.03800","n_code_links":0,"syntology":null},{"paper":"/paper/deep-reinforcement-learning-with-swin","title":"Deep Reinforcement Learning with Swin Transformers","date":"2022-06-30","arxiv_id":"2206.15269","n_code_links":1,"syntology":null},{"paper":"/paper/dna-proximal-policy-optimization-with-a-dual","title":"DNA: Proximal Policy Optimization with a Dual Network Architecture","date":"2022-06-20","arxiv_id":"2206.10027","n_code_links":1,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":72},{"task":"/task/q-learning","name":"Q-Learning","papers":63},{"task":"/task/reinforcement-learning-2","name":"reinforcement-learning","papers":61},{"task":"/task/reinforcement-learning","name":"Reinforcement Learning","papers":58},{"task":"/task/deep-reinforcement-learning","name":"Deep Reinforcement Learning","papers":47},{"task":"/task/atari-games","name":"Atari Games","papers":18},{"task":"/task/openai-gym","name":"OpenAI Gym","papers":10},{"task":"/task/decision-making","name":"Decision Making","papers":9},{"task":"/task/continuous-control","name":"Continuous Control","papers":7},{"task":"/task/continuous-control","name":"continuous-control","papers":6},{"task":"/task/mujoco","name":"MuJoCo","papers":5},{"task":"/task/management","name":"Management","papers":4},{"task":"/task/multi-agent-reinforcement-learning","name":"Multi-agent Reinforcement Learning","papers":4},{"task":"/task/scheduling","name":"Scheduling","papers":4},{"task":"/task/efficient-exploration","name":"Efficient Exploration","papers":3},{"task":"/task/general-reinforcement-learning","name":"General Reinforcement Learning","papers":3},{"task":"/task/traffic-signal-control","name":"Traffic Signal Control","papers":3},{"task":"/task/computational-efficiency","name":"Computational Efficiency","papers":2},{"task":"/task/diversity","name":"Diversity","papers":2},{"task":"/task/ensemble-learning","name":"Ensemble Learning","papers":2}],"tasks_shown":20,"n_tasks":67,"usage_by_year":[{"year":"2010","papers":1},{"year":"2015","papers":2},{"year":"2016","papers":2},{"year":"2017","papers":2},{"year":"2018","papers":8},{"year":"2019","papers":9},{"year":"2020","papers":27},{"year":"2021","papers":24},{"year":"2022","papers":14},{"year":"2023","papers":17},{"year":"2024","papers":4},{"year":"2025","papers":2}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/double-q-learning"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}