{"url":"/method/clipped-double-q-learning","slug":"clipped-double-q-learning","name":"Clipped Double Q-learning","full_name":"Clipped Double Q-learning","full_name_withheld":false,"description_markdown":"**Clipped Double Q-learning** is a variant on [Double Q-learning](https://paperswithcode.com/method/double-q-learning) that upper-bounds the less biased Q estimate $Q\\_{\\theta\\_{2}}$ by the biased estimate $Q\\_{\\theta\\_{1}}$. This is equivalent to taking the minimum of the two estimates, resulting in the following target update:\r\n\r\n$$ y\\_{1} = r + \\gamma\\min\\_{i=1,2}Q\\_{\\theta'\\_{i}}\\left(s', \\pi\\_{\\phi\\_{1}}\\left(s'\\right)\\right) $$\r\n\r\nThe motivation for this extension is that vanilla double [Q-learning](https://paperswithcode.com/method/q-learning) is sometimes ineffective if the target and current networks are too similar, e.g. with a slow-changing policy in an actor-critic framework.","description_state":"present","introduced_year":null,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":"http://arxiv.org/abs/1802.09477v3","title":"Addressing Function Approximation Error in Actor-Critic Methods","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Reinforcement Learning","area_id":"reinforcement-learning","collection":"Off-Policy TD Control","url":"/methods/category/off-policy-td-control","pwc_aliases":[]}],"n_papers_tagged":122,"archive_num_papers":null,"papers_newest_first":[{"paper":"/paper/gradual-transition-from-bellman-optimality","title":"Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning","date":"2025-06-06","arxiv_id":"2506.05968","n_code_links":1,"syntology":{"ran":1,"of":4,"unverified":3,"pointer_only":0}},{"paper":null,"title":"FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Control","date":"2025-05-28","arxiv_id":"2505.22642","n_code_links":0,"syntology":null},{"paper":null,"title":"LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models","date":"2025-05-21","arxiv_id":"2505.15293","n_code_links":0,"syntology":null},{"paper":null,"title":"Monte Carlo Beam Search for Actor-Critic Reinforcement Learning in Continuous Control","date":"2025-05-13","arxiv_id":"2505.09029","n_code_links":0,"syntology":null},{"paper":null,"title":"Energy Efficient RSMA-Based LEO Satellite Communications Assisted by UAV-Mounted BD-Active RIS: A DRL Approach","date":"2025-05-07","arxiv_id":"2505.04148","n_code_links":0,"syntology":null},{"paper":null,"title":"AlphaGrad: Non-Linear Gradient Normalization Optimizer","date":"2025-04-22","arxiv_id":"2504.16020","n_code_links":0,"syntology":null},{"paper":null,"title":"Motion Control in Multi-Rotor Aerial Robots Using Deep Reinforcement Learning","date":"2025-02-09","arxiv_id":"2502.05996","n_code_links":0,"syntology":null},{"paper":"/paper/td3-tucker-decomposition-based-dataset","title":"TD3: Tucker Decomposition Based Dataset Distillation Method for Sequential Recommendation","date":"2025-02-05","arxiv_id":"2502.02854","n_code_links":1,"syntology":null},{"paper":"/paper/evorl-a-gpu-accelerated-framework-for","title":"EvoRL: A GPU-accelerated Framework for Evolutionary Reinforcement Learning","date":"2025-01-25","arxiv_id":"2501.15129","n_code_links":2,"syntology":null},{"paper":null,"title":"Enhancing UAV Path Planning Efficiency Through Accelerated Learning","date":"2025-01-17","arxiv_id":"2501.10141","n_code_links":0,"syntology":null},{"paper":null,"title":"An Advantage-based Optimization Method for Reinforcement Learning in Large Action Space","date":"2024-12-17","arxiv_id":"2412.12605","n_code_links":0,"syntology":null},{"paper":null,"title":"Provably Efficient Action-Manipulation Attack Against Continuous Reinforcement Learning","date":"2024-11-20","arxiv_id":"2411.13116","n_code_links":0,"syntology":null},{"paper":"/paper/reinforcement-learning-gradients-as-vitamin","title":"Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers","date":"2024-10-31","arxiv_id":"2410.24108","n_code_links":1,"syntology":{"ran":0,"of":7,"unverified":7,"pointer_only":7}},{"paper":"/paper/networkgym-reinforcement-learning","title":"NetworkGym: Reinforcement Learning Environments for Multi-Access Traffic Management in Network Simulation","date":"2024-10-30","arxiv_id":"2411.04138","n_code_links":1,"syntology":{"ran":9,"of":18,"unverified":9,"pointer_only":0}},{"paper":"/paper/human-readable-programs-as-actors-of","title":"Human-Readable Programs as Actors of Reinforcement Learning Agents Using Critic-Moderated Evolution","date":"2024-10-29","arxiv_id":"2410.21940","n_code_links":1,"syntology":null},{"paper":null,"title":"Mitigating Suboptimality of Deterministic Policy Gradients in Complex Q-functions","date":"2024-10-15","arxiv_id":"2410.11833","n_code_links":0,"syntology":null},{"paper":null,"title":"Navigation in a simplified Urban Flow through Deep Reinforcement Learning","date":"2024-09-26","arxiv_id":"2409.17922","n_code_links":0,"syntology":null},{"paper":null,"title":"Simultaneous Training of First- and Second-Order Optimizers in Population-Based Reinforcement Learning","date":"2024-08-27","arxiv_id":"2408.15421","n_code_links":0,"syntology":null},{"paper":null,"title":"Optimizing TD3 for 7-DOF Robotic Arm Grasping: Overcoming Suboptimality with Exploration-Enhanced Contrastive Learning","date":"2024-08-26","arxiv_id":"2408.14009","n_code_links":0,"syntology":null},{"paper":null,"title":"Image-Based Deep Reinforcement Learning with Intrinsically Motivated Stimuli: On the Execution of Complex Robotic Tasks","date":"2024-07-31","arxiv_id":"2407.21338","n_code_links":0,"syntology":null},{"paper":null,"title":"Deep Reinforcement Learning Strategies in Finance: Insights into Asset Holding, Trading Behavior, and Purchase Diversity","date":"2024-06-29","arxiv_id":"2407.09557","n_code_links":0,"syntology":null},{"paper":null,"title":"Performance Comparison of Deep RL Algorithms for Mixed Traffic Cooperative Lane-Changing","date":"2024-06-25","arxiv_id":"2407.02521","n_code_links":0,"syntology":null},{"paper":null,"title":"Value Improved Actor Critic Algorithms","date":"2024-06-03","arxiv_id":"2406.01423","n_code_links":0,"syntology":null},{"paper":null,"title":"Chaos-based reinforcement learning with TD3","date":"2024-05-15","arxiv_id":"2405.09086","n_code_links":0,"syntology":null},{"paper":null,"title":"Offline Reinforcement Learning with Behavioral Supervisor Tuning","date":"2024-04-25","arxiv_id":"2404.16399","n_code_links":0,"syntology":null},{"paper":null,"title":"RLingua: Improving Reinforcement Learning Sample Efficiency in Robotic Manipulations With Large Language Models","date":"2024-03-11","arxiv_id":"2403.06420","n_code_links":0,"syntology":null},{"paper":null,"title":"Conservative DDPG -- Pessimistic RL without Ensemble","date":"2024-03-08","arxiv_id":"2403.05732","n_code_links":0,"syntology":null},{"paper":null,"title":"Fill-and-Spill: Deep Reinforcement Learning Policy Gradient Methods for Reservoir Operation Decision and Control","date":"2024-03-07","arxiv_id":"2403.04195","n_code_links":0,"syntology":null},{"paper":null,"title":"Exploiting Estimation Bias in Clipped Double Q-Learning for Continous Control Reinforcement Learning Tasks","date":"2024-02-14","arxiv_id":"2402.09078","n_code_links":0,"syntology":null},{"paper":null,"title":"MinMaxMin $Q$-learning","date":"2024-02-03","arxiv_id":"2402.05951","n_code_links":0,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":70},{"task":"/task/reinforcement-learning-2","name":"reinforcement-learning","papers":56},{"task":"/task/deep-reinforcement-learning","name":"Deep Reinforcement Learning","papers":55},{"task":"/task/reinforcement-learning","name":"Reinforcement Learning","papers":46},{"task":"/task/continuous-control","name":"Continuous Control","papers":32},{"task":"/task/continuous-control","name":"continuous-control","papers":31},{"task":"/task/mujoco","name":"MuJoCo","papers":27},{"task":"/task/q-learning","name":"Q-Learning","papers":17},{"task":"/task/openai-gym","name":"OpenAI Gym","papers":10},{"task":"/task/decision-making","name":"Decision Making","papers":8},{"task":"/task/autonomous-driving","name":"Autonomous Driving","papers":6},{"task":"/task/offline-rl","name":"Offline RL","papers":5},{"task":"/task/benchmarking","name":"Benchmarking","papers":4},{"task":"/task/meta-learning","name":"Meta-Learning","papers":4},{"task":"/task/policy-gradient-methods","name":"Policy Gradient Methods","papers":4},{"task":"/task/d4rl","name":"D4RL","papers":3},{"task":null,"name":"GPU","papers":3},{"task":"/task/scheduling","name":"Scheduling","papers":3},{"task":"/task/atari-games","name":"Atari Games","papers":2},{"task":"/task/computational-efficiency","name":"Computational Efficiency","papers":2}],"tasks_shown":20,"n_tasks":69,"usage_by_year":[{"year":"2018","papers":1},{"year":"2019","papers":6},{"year":"2020","papers":19},{"year":"2021","papers":24},{"year":"2022","papers":11},{"year":"2023","papers":26},{"year":"2024","papers":25},{"year":"2025","papers":10}],"row_source":"embedded","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/clipped-double-q-learning"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}