{"url":"/method/target-policy-smoothing","slug":"target-policy-smoothing","name":"Target Policy Smoothing","full_name":"Target Policy Smoothing","full_name_withheld":false,"description_markdown":"**Target Policy Smoothing** is a regularization strategy for the value function in reinforcement learning. Deterministic policies can overfit to narrow peaks in the value estimate, making them highly susceptible to functional approximation error, increasing the variance of the target. To reduce this variance, target policy smoothing adds a small amount of random noise to the target policy and averages over mini-batches - approximating a [SARSA](https://paperswithcode.com/method/sarsa)-like expectation/integral.\r\n\r\nThe modified target update is:\r\n\r\n$$ y = r + \\gamma{Q}\\_{\\theta'}\\left(s', \\pi\\_{\\theta'}\\left(s'\\right) + \\epsilon \\right) $$\r\n\r\n$$ \\epsilon \\sim \\text{clip}\\left(\\mathcal{N}\\left(0, \\sigma\\right), -c, c \\right) $$\r\n\r\nwhere the added noise is clipped to keep the target close to the original action. The outcome is an algorithm reminiscent of [Expected SARSA](https://paperswithcode.com/method/expected-sarsa), where the value estimate is instead learned off-policy and the noise added to the target policy is chosen independently of the exploration policy. The value estimate learned is with respect to a noisy policy defined by the parameter $\\sigma$.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Addressing Function Approximation Error in Actor-Critic Methods","paper":"/paper/addressing-function-approximation-error-in","first_author":"Scott Fujimoto","n_authors":3,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/addressing-function-approximation-error-in"},"source":{"url":"http://arxiv.org/abs/1802.09477v3","title":"Addressing Function Approximation Error in Actor-Critic Methods","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/sfujim/TD3/blob/ade6260da88864d1ab0ed592588e090d3d97d679/main.py#L86","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Regularization","url":"/methods/category/regularization","pwc_aliases":[]}],"n_papers_tagged":116,"archive_num_papers":116,"papers_newest_first":[{"paper":"/paper/gradual-transition-from-bellman-optimality","title":"Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning","date":"2025-06-06","arxiv_id":"2506.05968","n_code_links":1,"syntology":{"ran":1,"of":4,"unverified":3,"pointer_only":0}},{"paper":null,"title":"FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Control","date":"2025-05-28","arxiv_id":"2505.22642","n_code_links":0,"syntology":null},{"paper":null,"title":"LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models","date":"2025-05-21","arxiv_id":"2505.15293","n_code_links":0,"syntology":null},{"paper":null,"title":"Monte Carlo Beam Search for Actor-Critic Reinforcement Learning in Continuous Control","date":"2025-05-13","arxiv_id":"2505.09029","n_code_links":0,"syntology":null},{"paper":null,"title":"Energy Efficient RSMA-Based LEO Satellite Communications Assisted by UAV-Mounted BD-Active RIS: A DRL Approach","date":"2025-05-07","arxiv_id":"2505.04148","n_code_links":0,"syntology":null},{"paper":null,"title":"AlphaGrad: Non-Linear Gradient Normalization Optimizer","date":"2025-04-22","arxiv_id":"2504.16020","n_code_links":0,"syntology":null},{"paper":null,"title":"Motion Control in Multi-Rotor Aerial Robots Using Deep Reinforcement Learning","date":"2025-02-09","arxiv_id":"2502.05996","n_code_links":0,"syntology":null},{"paper":"/paper/td3-tucker-decomposition-based-dataset","title":"TD3: Tucker Decomposition Based Dataset Distillation Method for Sequential Recommendation","date":"2025-02-05","arxiv_id":"2502.02854","n_code_links":1,"syntology":null},{"paper":"/paper/evorl-a-gpu-accelerated-framework-for","title":"EvoRL: A GPU-accelerated Framework for Evolutionary Reinforcement Learning","date":"2025-01-25","arxiv_id":"2501.15129","n_code_links":2,"syntology":null},{"paper":null,"title":"Enhancing UAV Path Planning Efficiency Through Accelerated Learning","date":"2025-01-17","arxiv_id":"2501.10141","n_code_links":0,"syntology":null},{"paper":null,"title":"An Advantage-based Optimization Method for Reinforcement Learning in Large Action Space","date":"2024-12-17","arxiv_id":"2412.12605","n_code_links":0,"syntology":null},{"paper":null,"title":"Provably Efficient Action-Manipulation Attack Against Continuous Reinforcement Learning","date":"2024-11-20","arxiv_id":"2411.13116","n_code_links":0,"syntology":null},{"paper":"/paper/reinforcement-learning-gradients-as-vitamin","title":"Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers","date":"2024-10-31","arxiv_id":"2410.24108","n_code_links":1,"syntology":{"ran":0,"of":7,"unverified":7,"pointer_only":7}},{"paper":"/paper/networkgym-reinforcement-learning","title":"NetworkGym: Reinforcement Learning Environments for Multi-Access Traffic Management in Network Simulation","date":"2024-10-30","arxiv_id":"2411.04138","n_code_links":1,"syntology":{"ran":9,"of":18,"unverified":9,"pointer_only":0}},{"paper":"/paper/human-readable-programs-as-actors-of","title":"Human-Readable Programs as Actors of Reinforcement Learning Agents Using Critic-Moderated Evolution","date":"2024-10-29","arxiv_id":"2410.21940","n_code_links":1,"syntology":null},{"paper":null,"title":"Mitigating Suboptimality of Deterministic Policy Gradients in Complex Q-functions","date":"2024-10-15","arxiv_id":"2410.11833","n_code_links":0,"syntology":null},{"paper":null,"title":"Navigation in a simplified Urban Flow through Deep Reinforcement Learning","date":"2024-09-26","arxiv_id":"2409.17922","n_code_links":0,"syntology":null},{"paper":null,"title":"Simultaneous Training of First- and Second-Order Optimizers in Population-Based Reinforcement Learning","date":"2024-08-27","arxiv_id":"2408.15421","n_code_links":0,"syntology":null},{"paper":null,"title":"Optimizing TD3 for 7-DOF Robotic Arm Grasping: Overcoming Suboptimality with Exploration-Enhanced Contrastive Learning","date":"2024-08-26","arxiv_id":"2408.14009","n_code_links":0,"syntology":null},{"paper":null,"title":"Image-Based Deep Reinforcement Learning with Intrinsically Motivated Stimuli: On the Execution of Complex Robotic Tasks","date":"2024-07-31","arxiv_id":"2407.21338","n_code_links":0,"syntology":null},{"paper":null,"title":"Deep Reinforcement Learning Strategies in Finance: Insights into Asset Holding, Trading Behavior, and Purchase Diversity","date":"2024-06-29","arxiv_id":"2407.09557","n_code_links":0,"syntology":null},{"paper":null,"title":"Performance Comparison of Deep RL Algorithms for Mixed Traffic Cooperative Lane-Changing","date":"2024-06-25","arxiv_id":"2407.02521","n_code_links":0,"syntology":null},{"paper":null,"title":"Value Improved Actor Critic Algorithms","date":"2024-06-03","arxiv_id":"2406.01423","n_code_links":0,"syntology":null},{"paper":null,"title":"Chaos-based reinforcement learning with TD3","date":"2024-05-15","arxiv_id":"2405.09086","n_code_links":0,"syntology":null},{"paper":null,"title":"Offline Reinforcement Learning with Behavioral Supervisor Tuning","date":"2024-04-25","arxiv_id":"2404.16399","n_code_links":0,"syntology":null},{"paper":null,"title":"RLingua: Improving Reinforcement Learning Sample Efficiency in Robotic Manipulations With Large Language Models","date":"2024-03-11","arxiv_id":"2403.06420","n_code_links":0,"syntology":null},{"paper":null,"title":"Conservative DDPG -- Pessimistic RL without Ensemble","date":"2024-03-08","arxiv_id":"2403.05732","n_code_links":0,"syntology":null},{"paper":null,"title":"Fill-and-Spill: Deep Reinforcement Learning Policy Gradient Methods for Reservoir Operation Decision and Control","date":"2024-03-07","arxiv_id":"2403.04195","n_code_links":0,"syntology":null},{"paper":null,"title":"Exploiting Estimation Bias in Clipped Double Q-Learning for Continous Control Reinforcement Learning Tasks","date":"2024-02-14","arxiv_id":"2402.09078","n_code_links":0,"syntology":null},{"paper":null,"title":"MinMaxMin $Q$-learning","date":"2024-02-03","arxiv_id":"2402.05951","n_code_links":0,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":67},{"task":"/task/reinforcement-learning-2","name":"reinforcement-learning","papers":54},{"task":"/task/deep-reinforcement-learning","name":"Deep Reinforcement Learning","papers":53},{"task":"/task/reinforcement-learning","name":"Reinforcement Learning","papers":46},{"task":"/task/continuous-control","name":"Continuous Control","papers":30},{"task":"/task/continuous-control","name":"continuous-control","papers":29},{"task":"/task/mujoco","name":"MuJoCo","papers":25},{"task":"/task/q-learning","name":"Q-Learning","papers":11},{"task":"/task/openai-gym","name":"OpenAI Gym","papers":9},{"task":"/task/decision-making","name":"Decision Making","papers":8},{"task":"/task/autonomous-driving","name":"Autonomous Driving","papers":6},{"task":"/task/offline-rl","name":"Offline RL","papers":5},{"task":"/task/benchmarking","name":"Benchmarking","papers":4},{"task":"/task/meta-learning","name":"Meta-Learning","papers":4},{"task":"/task/policy-gradient-methods","name":"Policy Gradient Methods","papers":4},{"task":"/task/d4rl","name":"D4RL","papers":3},{"task":null,"name":"GPU","papers":3},{"task":"/task/scheduling","name":"Scheduling","papers":3},{"task":"/task/atari-games","name":"Atari Games","papers":2},{"task":"/task/computational-efficiency","name":"Computational Efficiency","papers":2}],"tasks_shown":20,"n_tasks":69,"usage_by_year":[{"year":"2018","papers":1},{"year":"2019","papers":6},{"year":"2020","papers":18},{"year":"2021","papers":21},{"year":"2022","papers":10},{"year":"2023","papers":25},{"year":"2024","papers":25},{"year":"2025","papers":10}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/target-policy-smoothing"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}