Methods › Reinforcement Learning › Off-Policy TD Control › Double Q-learning
Double Q-learning
Introduced by Hado V. Hasselt in Double Q-learning
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Double Q-learning is an off-policy reinforcement learning algorithm that utilises double estimation to counteract overestimation problems with traditional Q-learning.
The max operator in standard Q-learning and DQN uses the same values both to select and to evaluate an action. This makes it more likely to select overestimated values, resulting in overoptimistic value estimates. To prevent this, we can decouple the selection from the evaluation, which is the idea behind Double Q-learning:
Y^Qₜ = Rₜ₊₁ + γQ(Sₜ₊₁, argmaxₐQ(Sₜ₊₁, a; θₜ);θₜ)
The Double Q-learning error can then be written as:
Y^(DoubleQ)ₜ = Rₜ₊₁ + γQ(Sₜ₊₁, argmaxₐQ(Sₜ₊₁, a; θₜ);θ^’ₜ)
Here the selection of the action in the argmax is still due to the online weights θₜ. But we use a second set of weights θ^’ₜ to fairly evaluate the value of this policy.
In Syntology for a different name, 'DOUBLE': repository known, no verified implementation (declared_code; repositories: 1 listed of 1 in the graph, strongest link: listed only). The lookup matched a different name in Syntology's graph, 'DOUBLE' (resolution: token_match); this answer is about that name. The archive credits no paper with an arXiv id for this method, so whether this answer (about arXiv:2601.05524) is about this method is not checked.
Code Syntology holds under other names, as the lookup lists them (code_nearby); not a complete list, and none of them is this method.
- Double-Restart Q-UCB · shares_name_tokens (shares name tokens with this name; a different method) · level V2 · retrieve with
get_reference_implementation(name='Double-Restart Q-UCB') - Periodic Double Oracle (PDO) · shares_name_tokens (shares name tokens with this name; a different method) · level V3 · retrieve with
get_reference_implementation(name='Periodic Double Oracle (PDO)') - Randomized Ensembled Double Q-Learning (REDQ) · shares_name_tokens (shares name tokens with this name; a different method) · level V3 · retrieve with
get_reference_implementation(name='Randomized Ensembled Double Q-Learning (REDQ)')
Origin paper of this answer, arXiv:2601.05524 (its page on this site · its page on Syntology): in Syntology's graph layer with no harvested code sample.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
What each level means, quoted from Syntology's server card: V0 “imports”; V1 “executes against a synthesized fixture and returns a value of the recorded type”; V2 “two implementations GENERATED from the paper's signature and spec agree numerically (1e-6) on fixture inputs” “neither of the two ever read the authors' own code, so V2 measures how unambiguously the paper specifies the method, not fidelity to the authors' implementation (measured 2026-09-11, commit 75eaa77: cross_checked_by is a second LLM on 2,628 of 2,831 served samples)”; V3 “paper-derived property tests pass on both of those same two generations.” The card adds: “No rung on this ladder compares the served code against the authors' implementation, so treat every level as evidence about runnability and internal agreement, not about matching the paper's results.” A level quoted in two parts is one definition the card writes as two clauses joined by a double hyphen, each clause quoted on its own (how these levels are quoted).
Papers archive 2025-07-28
archive papers tagged: 112 · with a code link: 45 · where Syntology ran a sample: 15 (13 with a run with no instrument failure, 2 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (15 of 112 tagged: 13 with a run with no instrument failure, 2 where every run was a failure of Syntology's instrument)
Page 1 of 2: papers 1 to 100 of 112, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
Reinforcement Learning-Based Policy Optimisation For Heterogeneous Radio Access 18 Jun 2025 · 0 repositories · arXiv:2506.15273
-
Dynamic Operating System Scheduling Using Double DQN: A Reinforcement Learning Approach to Task Optimization 31 Mar 2025 · 0 repositories · arXiv:2503.23659
-
Distribution-Free Uncertainty Quantification in Mechanical Ventilation Treatment: A Conformal Deep Q-Learning Framework 17 Dec 2024 · 0 repositories · arXiv:2412.12597
-
Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC 6 Nov 2024 · 3 repositories · arXiv:2411.03820Syntology official (archive's flag): 3 ran · 16 ran (of which 10 constructed an object rather than computing a result; 12 with no instrument failure: 0 honoured, 0 violated, 12 with no contract checked; 4 where Syntology's instrument failed) · 9 unverified (of 25 harvested samples) · 21 pointer-only (licence)
-
Bootstrapping Expectiles in Reinforcement Learning 6 Jun 2024 · 0 repositories · arXiv:2406.04081
-
A New View on Planning in Online Reinforcement Learning 3 Jun 2024 · 0 repositories · arXiv:2406.01562
-
Active search and coverage using point-cloud reinforcement learning 18 Dec 2023 · 0 repositories · arXiv:2312.11410
-
Efficient Sparse-Reward Goal-Conditioned Reinforcement Learning with a High Replay Ratio and Regularization 10 Dec 2023 · 1 repository · arXiv:2312.05787
-
Data-efficient Deep Reinforcement Learning for Vehicle Trajectory Control 30 Nov 2023 · 0 repositories · arXiv:2311.18393
-
Advancing Algorithmic Trading: A Multi-Technique Enhancement of Deep Q-Network Models 9 Nov 2023 · 0 repositories · arXiv:2311.05743
-
Deep Reinforcement Learning for the Heat Transfer Control of Pulsating Impinging Jets 25 Sep 2023 · 0 repositories · arXiv:2309.13955
-
Adaptive Multi-Agent Deep Reinforcement Learning for Timely Healthcare Interventions 20 Sep 2023 · 0 repositories · arXiv:2309.10980
-
Deep Reinforcement Learning for Artificial Upwelling Energy Management 20 Aug 2023 · 0 repositories · arXiv:2308.10199
-
Interpretable and Secure Trajectory Optimization for UAV-Assisted Communication 5 Jul 2023 · 0 repositories · arXiv:2307.02002
-
Optimizing Credit Limit Adjustments Under Adversarial Goals Using Reinforcement Learning 27 Jun 2023 · 0 repositories · arXiv:2306.15585
-
Vanishing Bias Heuristic-guided Reinforcement Learning Algorithm 17 Jun 2023 · 0 repositories · arXiv:2306.10216
-
RSRM: Reinforcement Symbolic Regression Machine 24 May 2023 · 0 repositories · arXiv:2305.14656
-
Extracting Diagnosis Pathways from Electronic Health Records Using Deep Reinforcement Learning 10 May 2023 · 1 repository · arXiv:2305.06295
-
Train a Real-world Local Path Planner in One Hour via Partially Decoupled Reinforcement Learning and Vectorized Diversity 7 May 2023 · 1 repository · arXiv:2305.04180
-
Smoothed Q-learning 15 Mar 2023 · 0 repositories · arXiv:2303.08631
-
Double A3C: Deep Reinforcement Learning on OpenAI Gym Games 4 Mar 2023 · 0 repositories · arXiv:2303.02271
-
Asynchronous Deep Double Duelling Q-Learning for Trading-Signal Execution in Limit Order Book Markets 20 Jan 2023 · 0 repositories · arXiv:2301.08688
-
Hierarchical Deep Q-Learning Based Handover in Wireless Networks with Dual Connectivity 13 Jan 2023 · 0 repositories · arXiv:2301.05391
-
Neighboring state-based RL Exploration 21 Dec 2022 · 0 repositories · arXiv:2212.10712
-
Elastic Step DQN: A novel multi-step algorithm to alleviate overestimation in Deep QNetworks 7 Oct 2022 · 0 repositories · arXiv:2210.03325
-
M²DQN: A Robust Method for Accelerating Deep Q-learning Network 16 Sep 2022 · 1 repository · arXiv:2209.07809
-
Reducing Variance in Temporal-Difference Value Estimation via Ensemble of Deep Networks 16 Sep 2022 · 1 repository · arXiv:2209.07670
-
Double Q-Learning for Citizen Relocation During Natural Hazards 8 Sep 2022 · 0 repositories · arXiv:2209.03800
-
Deep Reinforcement Learning with Swin Transformers 30 Jun 2022 · 1 repository · arXiv:2206.15269
-
DNA: Proximal Policy Optimization with a Dual Network Architecture 20 Jun 2022 · 1 repository · arXiv:2206.10027
-
Goal-Space Planning with Subgoal Models 6 Jun 2022 · 0 repositories · arXiv:2206.02902
-
Simultaneous Double Q-learning with Conservative Advantage Learning for Actor-Critic Methods 8 May 2022 · 1 repository · arXiv:2205.03819
-
Optimizing the Long-Term Behaviour of Deep Reinforcement Learning for Pushing and Grasping 7 Apr 2022 · 0 repositories · arXiv:2204.03487
-
Action Candidate Driven Clipped Double Q-learning for Discrete and Continuous Action Tasks 22 Mar 2022 · 1 repository · arXiv:2203.11526
-
Random Ensemble Reinforcement Learning for Traffic Signal Control 10 Mar 2022 · 0 repositories · arXiv:2203.05961
-
skrl: Modular and Flexible Library for Reinforcement Learning 8 Feb 2022 · 1 repository · arXiv:2202.03825
-
An Improved Reinforcement Learning Algorithm for Learning to Branch 17 Jan 2022 · 0 repositories · arXiv:2201.06213
-
Intelligent Traffic Light via Policy-based Deep Reinforcement Learning 27 Dec 2021 · 1 repository · arXiv:2112.13817
-
Faster Non-asymptotic Convergence for Double Q-learning 1 Dec 2021 · 0 repositories
-
Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performance 17 Nov 2021 · 0 repositories · arXiv:2111.09159
-
A study of first-passage time minimization via Q-learning in heated gridworlds 5 Oct 2021 · 0 repositories · arXiv:2110.02129
-
Dropout Q-Functions for Doubly Efficient Reinforcement Learning 5 Oct 2021 · 2 repositories · arXiv:2110.02034Syntology official (archive's flag): 1 ran · 6 ran (of which 3 constructed an object rather than computing a result; 6 with no instrument failure: 2 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (of 8 harvested samples)
-
On the Estimation Bias in Double Q-Learning 29 Sep 2021 · 1 repository · arXiv:2109.14419Syntology official (archive's flag): 4 ran · 4 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified (of 6 harvested samples)
-
Parameter-free Reduction of the Estimation Bias in Deep Reinforcement Learning for Deterministic Policy Gradients 24 Sep 2021 · 1 repository · arXiv:2109.11788
-
Estimation Error Correction in Deep Reinforcement Learning for Deterministic Actor-Critic Methods 22 Sep 2021 · 1 repository · arXiv:2109.10736
-
Deep Reinforcement Learning at the Edge of the Statistical Precipice 30 Aug 2021 · 3 repositories · arXiv:2108.13264Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 4 honoured, 1 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 5 harvested samples)
-
Modified Double DQN: addressing stability 9 Aug 2021 · 0 repositories · arXiv:2108.04115
-
An Improved Algorithm of Robot Path Planning in Complex Environment Based on Double DQN 23 Jul 2021 · 0 repositories · arXiv:2107.11245
-
A Reinforcement Learning Environment for Mathematical Reasoning via Program Synthesis 15 Jul 2021 · 1 repository · arXiv:2107.07373
-
Transfer Learning in Multi-Agent Reinforcement Learning with Double Q-Networks for Distributed Resource Sharing in V2X Communication 13 Jul 2021 · 0 repositories · arXiv:2107.06195
-
Expert Q-learning: Deep Reinforcement Learning with Coarse State Values from Offline Expert Examples 28 Jun 2021 · 0 repositories · arXiv:2106.14642
-
Learning to Optimize Industry-Scale Dynamic Pickup and Delivery Problems 27 May 2021 · 0 repositories · arXiv:2105.12899
-
Action Candidate Based Clipped Double Q-learning for Discrete and Continuous Action Tasks 3 May 2021 · 1 repository · arXiv:2105.00704
-
Independent Reinforcement Learning for Weakly Cooperative Multiagent Traffic Control Problem 22 Apr 2021 · 1 repository · arXiv:2104.10917
-
A coevolutionary approach to deep multi-agent reinforcement learning 12 Apr 2021 · 1 repository · arXiv:2104.05610
-
Variational quantum compiling with double Q-learning 22 Mar 2021 · 0 repositories · arXiv:2103.11611
-
Greedy-Step Off-Policy Reinforcement Learning 23 Feb 2021 · 0 repositories · arXiv:2102.11717
-
Adaptive Rational Activations to Boost Deep Reinforcement Learning 18 Feb 2021 · 4 repositories · arXiv:2102.09407Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 6 where Syntology's instrument failed) · 0 unverified (of 6 harvested samples) · 3 pointer-only (licence)
-
Randomized Ensembled Double Q-Learning: Learning Fast Without a Model 15 Jan 2021 · 6 repositories · arXiv:2101.05982Syntology community repositories only · 4 ran (of which 4 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified; every one of the 4 samples that ran constructed an object rather than computing a result (of 5 harvested samples) · 3 pointer-only (licence)
-
Double Q-learning: New Analysis and Sharper Finite-time Bound 1 Jan 2021 · 0 repositories
-
Weighted Bellman Backups for Improved Signal-to-Noise in Q-Updates 1 Jan 2021 · 0 repositories
-
A State Representation Dueling Network for Deep Reinforcement Learning 24 Dec 2020 · 0 repositories
-
Mobile Robots Autonomous Exploration with Reinforcement Learning 14 Dec 2020 · 0 repositories
-
Resolving Implicit Coordination in Multi-Agent Deep Reinforcement Learning with Deep Q-Networks & Game Theory 8 Dec 2020 · 1 repository · arXiv:2012.09136
-
Self-correcting Q-Learning 2 Dec 2020 · 0 repositories · arXiv:2012.01100
-
Energy and Service-priority aware Trajectory Design for UAV-BSs using Double Q-Learning 26 Oct 2020 · 0 repositories · arXiv:2010.13346
-
Finite-Time Analysis for Double Q-learning 29 Sep 2020 · 0 repositories · arXiv:2009.14257
-
A New Approach for Tactical Decision Making in Lane Changing: Sample Efficient Deep Q Learning with a Safety Feedback Reward 24 Sep 2020 · 0 repositories · arXiv:2009.11905
-
Chrome Dino Run using Reinforcement Learning 15 Aug 2020 · 0 repositories · arXiv:2008.06799
-
Reinforcement Learning with Quantum Variational Circuits 15 Aug 2020 · 2 repositories · arXiv:2008.07524
-
QPLEX: Duplex Dueling Multi-Agent Q-Learning 3 Aug 2020 · 6 repositories · arXiv:2008.01062Syntology community repositories only · 4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 4 harvested samples) · 1 pointer-only (licence)
-
The Mean-Squared Error of Double Q-Learning 9 Jul 2020 · 1 repository · arXiv:2007.05034Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples)
-
SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning 9 Jul 2020 · 1 repository · arXiv:2007.04938
-
Regularly Updated Deterministic Policy Gradient Algorithm 1 Jul 2020 · 0 repositories · arXiv:2007.00169
-
Some approaches used to overcome overestimation in Deep Reinforcement Learning algorithms 25 Jun 2020 · 0 repositories · arXiv:2006.14167
-
Deep Reinforcement Learning Control for Radar Detection and Tracking in Congested Spectral Environments 23 Jun 2020 · 0 repositories · arXiv:2006.13173
-
Parameterized MDPs and Reinforcement Learning Problems -- A Maximum Entropy Principle Based Framework 17 Jun 2020 · 0 repositories · arXiv:2006.09646
-
Learning Heuristic Selection with Dynamic Algorithm Configuration 15 Jun 2020 · 1 repository · arXiv:2006.08246
-
Decorrelated Double Q-learning 12 Jun 2020 · 0 repositories · arXiv:2006.06956
-
Privacy-Cost Management in Smart Meters with Mutual Information-Based Reinforcement Learning 10 Jun 2020 · 0 repositories · arXiv:2006.06106
-
Balancing a CartPole System with Reinforcement Learning -- A Tutorial 8 Jun 2020 · 0 repositories · arXiv:2006.04938
-
Acme: A Research Framework for Distributed Reinforcement Learning 1 Jun 2020 · 5 repositories · arXiv:2006.00979
-
Dynamic Algorithm Configuration: Foundation of a New Meta-Algorithmic Framework 1 Jun 2020 · 1 repository
-
Basal Glucose Control in Type 1 Diabetes using Deep Reinforcement Learning: An In Silico Validation 18 May 2020 · 0 repositories · arXiv:2005.09059
-
A Double Q-Learning Approach for Navigation of Aerial Vehicles with Connectivity Constraint 24 Feb 2020 · 0 repositories · arXiv:2002.10563
-
Disentangling Controllable Object through Video Prediction Improves Visual Reinforcement Learning 21 Feb 2020 · 0 repositories · arXiv:2002.09136
-
Fast Reinforcement Learning for Anti-jamming Communications 13 Feb 2020 · 0 repositories · arXiv:2002.05364
-
Regret Bounds for Discounted MDPs 12 Feb 2020 · 0 repositories · arXiv:2002.05138
-
SLM Lab: A Comprehensive Benchmark and Modular Software Framework for Reproducible Deep Reinforcement Learning 28 Dec 2019 · 1 repository · arXiv:1912.12482Syntology official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified (of 13 harvested samples)
-
Exploiting the potential of deep reinforcement learning for classification tasks in high-dimensional and unstructured data 20 Dec 2019 · 0 repositories · arXiv:1912.09595
-
Task-Oriented Language Grounding for Language Input with Multiple Sub-Goals of Non-Linear Order 27 Oct 2019 · 1 repository · arXiv:1910.12354
-
Reverse Experience Replay 19 Oct 2019 · 0 repositories · arXiv:1910.08780
-
To Combine or Not To Combine? A Rainbow Deep Reinforcement Learning Agent for Dialog Policies 1 Sep 2019 · 0 repositories
-
Performing Deep Recurrent Double Q-Learning for Atari Games 16 Aug 2019 · 2 repositories · arXiv:1908.06040
-
Large-Scale Traffic Signal Control Using a Novel Multi-Agent Reinforcement Learning 10 Aug 2019 · 0 repositories · arXiv:1908.03761
-
A Story of Two Streams: Reinforcement Learning Models from Human Behavior and Neuropsychiatry 21 Jun 2019 · 1 repository · arXiv:1906.11286
-
Generative Adversarial Imagination for Sample Efficient Deep Reinforcement Learning 30 Apr 2019 · 0 repositories · arXiv:1904.13255
-
Double Deep Q-Learning for Optimal Execution 17 Dec 2018 · 0 repositories · arXiv:1812.06600
-
Macro action selection with deep reinforcement learning in StarCraft 2 Dec 2018 · 1 repository · arXiv:1812.00336
-
Revisiting the Softmax Bellman Operator: New Benefits and New Perspective 2 Dec 2018 · 2 repositories · arXiv:1812.00456Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample) · 1 pointer-only (licence)
Tasks archive 2025-07-28
20 shown of 67 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections