{"url":"/method/trpo","slug":"trpo","name":"TRPO","full_name":"Trust Region Policy Optimization","full_name_withheld":false,"description_markdown":"**Trust Region Policy Optimization**, or **TRPO**, is a policy gradient method in reinforcement learning that avoids parameter updates that change the policy too much with a KL divergence constraint on the size of the policy update at each iteration.\r\n\r\nTake the case of off-policy reinforcement learning, where the policy $\\beta$ for collecting trajectories on rollout workers is different from the policy $\\pi$ to optimize for. The objective function in an off-policy model measures the total advantage over the state visitation distribution and actions, while the mismatch between the training data distribution and the true policy state distribution is compensated with an importance sampling estimator:\r\n\r\n$$ J\\left(\\theta\\right) = \\sum\\_{s\\in{S}}p^{\\pi\\_{\\theta\\_{old}}}\\sum\\_{a\\in\\mathcal{A}}\\left(\\pi\\_{\\theta}\\left(a\\mid{s}\\right)\\hat{A}\\_{\\theta\\_{old}}\\left(s, a\\right)\\right) $$\r\n\r\n$$ J\\left(\\theta\\right) = \\sum\\_{s\\in{S}}p^{\\pi\\_{\\theta\\_{old}}}\\sum\\_{a\\in\\mathcal{A}}\\left(\\beta\\left(a\\mid{s}\\right)\\frac{\\pi\\_{\\theta}\\left(a\\mid{s}\\right)}{\\beta\\left(a\\mid{s}\\right)}\\hat{A}\\_{\\theta\\_{old}}\\left(s, a\\right)\\right) $$\r\n\r\n$$ J\\left(\\theta\\right) = \\mathbb{E}\\_{s\\sim{p}^{\\pi\\_{\\theta\\_{old}}}, a\\sim{\\beta}} \\left(\\frac{\\pi\\_{\\theta}\\left(a\\mid{s}\\right)}{\\beta\\left(a\\mid{s}\\right)}\\hat{A}\\_{\\theta\\_{old}}\\left(s, a\\right)\\right)$$\r\n\r\nWhen training on policy, theoretically the policy for collecting data is same as the policy that we want to optimize. However, when rollout workers and optimizers are running in parallel asynchronously, the behavior policy can get stale. TRPO considers this subtle difference: It labels the behavior policy as $\\pi\\_{\\theta\\_{old}}\\left(a\\mid{s}\\right)$ and thus the objective function becomes:\r\n\r\n$$ J\\left(\\theta\\right) = \\mathbb{E}\\_{s\\sim{p}^{\\pi\\_{\\theta\\_{old}}}, a\\sim{\\pi\\_{\\theta\\_{old}}}} \\left(\\frac{\\pi\\_{\\theta}\\left(a\\mid{s}\\right)}{\\pi\\_{\\theta\\_{old}}\\left(a\\mid{s}\\right)}\\hat{A}\\_{\\theta\\_{old}}\\left(s, a\\right)\\right)$$\r\n\r\nTRPO aims to maximize the objective function $J\\left(\\theta\\right)$ subject to a trust region constraint which enforces the distance between old and new policies measured by KL-divergence to be small enough, within a parameter $\\delta$:\r\n\r\n$$ \\mathbb{E}\\_{s\\sim{p}^{\\pi\\_{\\theta\\_{old}}}} \\left[D\\_{KL}\\left(\\pi\\_{\\theta\\_{old}}\\left(.\\mid{s}\\right)\\mid\\mid\\pi\\_{\\theta}\\left(.\\mid{s}\\right)\\right)\\right] \\leq \\delta$$","description_state":"present","introduced_year":null,"introduced_by":{"title":"Trust Region Policy Optimization","paper":"/paper/trust-region-policy-optimization","first_author":"John Schulman","n_authors":5,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/trust-region-policy-optimization"},"source":{"url":"http://arxiv.org/abs/1502.05477v5","title":"Trust Region Policy Optimization","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Reinforcement Learning","area_id":"reinforcement-learning","collection":"Policy Gradient Methods","url":"/methods/category/policy-gradient-methods","pwc_aliases":[]}],"n_papers_tagged":81,"archive_num_papers":81,"papers_newest_first":[{"paper":null,"title":"StaQ it! Growing neural networks for Policy Mirror Descent","date":"2025-06-16","arxiv_id":"2506.13862","n_code_links":0,"syntology":null},{"paper":null,"title":"Finite-Sample Convergence Bounds for Trust Region Policy Optimization in Mean-Field Games","date":"2025-05-28","arxiv_id":"2505.22781","n_code_links":0,"syntology":null},{"paper":"/paper/improving-value-estimation-critically","title":"Improving Value Estimation Critically Enhances Vanilla Policy Gradient","date":"2025-05-25","arxiv_id":"2505.19247","n_code_links":1,"syntology":null},{"paper":null,"title":"Energy Efficient RSMA-Based LEO Satellite Communications Assisted by UAV-Mounted BD-Active RIS: A DRL Approach","date":"2025-05-07","arxiv_id":"2505.04148","n_code_links":0,"syntology":null},{"paper":null,"title":"Deep Reinforcement Learning-Based User Association in Hybrid LiFi/WiFi Indoor Networks","date":"2025-03-03","arxiv_id":"2503.01803","n_code_links":0,"syntology":null},{"paper":null,"title":"Fast Convergence of Softmax Policy Mirror Ascent","date":"2024-11-18","arxiv_id":"2411.12042","n_code_links":0,"syntology":null},{"paper":null,"title":"Dynamics of Resource Allocation in O-RANs: An In-depth Exploration of On-Policy and Off-Policy Deep Reinforcement Learning for Real-Time Applications","date":"2024-11-17","arxiv_id":"2412.01839","n_code_links":0,"syntology":null},{"paper":null,"title":"Embedding Safety into RL: A New Take on Trust Region Methods","date":"2024-11-05","arxiv_id":"2411.02957","n_code_links":0,"syntology":null},{"paper":null,"title":"Linear Function Approximation as a Computationally Efficient Method to Solve Classical Reinforcement Learning Challenges","date":"2024-05-27","arxiv_id":"2405.20350","n_code_links":0,"syntology":null},{"paper":"/paper/matrix-low-rank-trust-region-policy","title":"Matrix Low-Rank Trust Region Policy Optimization","date":"2024-05-27","arxiv_id":"2405.17625","n_code_links":1,"syntology":null},{"paper":"/paper/joint-physical-digital-facial-attack","title":"Joint Physical-Digital Facial Attack Detection Via Simulating Spoofing Clues","date":"2024-04-12","arxiv_id":"2404.08450","n_code_links":3,"syntology":null},{"paper":"/paper/policy-mirror-descent-with-lookahead","title":"Policy Mirror Descent with Lookahead","date":"2024-03-21","arxiv_id":"2403.14156","n_code_links":1,"syntology":null},{"paper":null,"title":"Convergence for Natural Policy Gradient on Infinite-State Queueing MDPs","date":"2024-02-07","arxiv_id":"2402.05274","n_code_links":0,"syntology":null},{"paper":"/paper/simple-policy-optimization","title":"Simple Policy Optimization","date":"2024-01-29","arxiv_id":"2401.16025","n_code_links":1,"syntology":null},{"paper":"/paper/clipped-objective-policy-gradients-for","title":"Clipped-Objective Policy Gradients for Pessimistic Policy Optimization","date":"2023-11-10","arxiv_id":"2311.05846","n_code_links":1,"syntology":null},{"paper":null,"title":"Dropout Strategy in Reinforcement Learning: Limiting the Surrogate Objective Variance in Policy Optimization Methods","date":"2023-10-31","arxiv_id":"2310.20380","n_code_links":0,"syntology":null},{"paper":"/paper/dsac-t-distributional-soft-actor-critic-with","title":"Distributional Soft Actor-Critic with Three Refinements","date":"2023-10-09","arxiv_id":"2310.05858","n_code_links":2,"syntology":{"ran":5,"of":5,"unverified":0,"pointer_only":5}},{"paper":null,"title":"General Munchausen Reinforcement Learning with Tsallis Kullback-Leibler Divergence","date":"2023-09-21","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":null,"title":"Distributional Estimation of Data Uncertainty for Surveillance Face Anti-spoofing","date":"2023-09-18","arxiv_id":"2309.09485","n_code_links":0,"syntology":null},{"paper":"/paper/containergym-a-real-world-reinforcement","title":"ContainerGym: A Real-World Reinforcement Learning Benchmark for Resource Allocation","date":"2023-07-06","arxiv_id":"2307.02991","n_code_links":1,"syntology":null},{"paper":null,"title":"Safe, Efficient, Comfort, and Energy-saving Automated Driving through Roundabout Based on Deep Reinforcement Learning","date":"2023-06-20","arxiv_id":"2306.11465","n_code_links":0,"syntology":null},{"paper":"/paper/decision-aware-actor-critic-with-function-1","title":"Decision-Aware Actor-Critic with Function Approximation and Theoretical Guarantees","date":"2023-05-24","arxiv_id":"2305.15249","n_code_links":1,"syntology":{"ran":1,"of":2,"unverified":1,"pointer_only":2}},{"paper":null,"title":"Trust-Region-Free Policy Optimization for Stochastic Policies","date":"2023-02-15","arxiv_id":"2302.07985","n_code_links":0,"syntology":null},{"paper":"/paper/sample-dropout-a-simple-yet-effective","title":"Sample Dropout: A Simple yet Effective Variance Reduction Technique in Deep Policy Optimization","date":"2023-02-05","arxiv_id":"2302.02299","n_code_links":1,"syntology":{"ran":3,"of":12,"unverified":9,"pointer_only":12}},{"paper":"/paper/learning-fast-and-slow-a-goal-directed-memory","title":"Learning, Fast and Slow: A Goal-Directed Memory-Based Approach for Dynamic Environments","date":"2023-01-31","arxiv_id":"2301.13758","n_code_links":1,"syntology":null},{"paper":"/paper/a-novel-framework-for-policy-mirror-descent","title":"A Novel Framework for Policy Mirror Descent with General Parameterization and Linear Convergence","date":"2023-01-30","arxiv_id":"2301.13139","n_code_links":1,"syntology":null},{"paper":null,"title":"Generalized Munchausen Reinforcement Learning using Tsallis KL Divergence","date":"2023-01-27","arxiv_id":"2301.11476","n_code_links":0,"syntology":null},{"paper":null,"title":"PDVN: A Patch-based Dual-view Network for Face Liveness Detection using Light Field Focal Stack","date":"2023-01-17","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/relative-sparsity-for-medical-decision","title":"Relative Sparsity for Medical Decision Problems","date":"2022-11-29","arxiv_id":"2211.16566","n_code_links":1,"syntology":null},{"paper":null,"title":"Normality-Guided Distributional Reinforcement Learning for Continuous Control","date":"2022-08-28","arxiv_id":"2208.13125","n_code_links":0,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":46},{"task":"/task/reinforcement-learning","name":"Reinforcement Learning","papers":32},{"task":"/task/reinforcement-learning-2","name":"reinforcement-learning","papers":30},{"task":"/task/deep-reinforcement-learning","name":"Deep Reinforcement Learning","papers":23},{"task":"/task/continuous-control","name":"Continuous Control","papers":11},{"task":"/task/policy-gradient-methods","name":"Policy Gradient Methods","papers":9},{"task":"/task/mujoco","name":"MuJoCo","papers":8},{"task":"/task/continuous-control","name":"continuous-control","papers":8},{"task":"/task/decision-making","name":"Decision Making","papers":5},{"task":"/task/atari-games","name":"Atari Games","papers":3},{"task":"/task/face-anti-spoofing","name":"Face Anti-Spoofing","papers":3},{"task":"/task/face-recognition","name":"Face Recognition","papers":3},{"task":"/task/multi-task-learning","name":"Multi-Task Learning","papers":3},{"task":"/task/benchmarking","name":"Benchmarking","papers":2},{"task":"/task/meta-learning","name":"Meta-Learning","papers":2},{"task":"/task/model-based-reinforcement-learning","name":"Model-based Reinforcement Learning","papers":2},{"task":"/task/openai-gym","name":"OpenAI Gym","papers":2},{"task":"/task/partially-observable-reinforcement-learning","name":"Partially Observable Reinforcement Learning","papers":2},{"task":"/task/problem-decomposition","name":"Problem Decomposition","papers":2},{"task":"/task/q-learning","name":"Q-Learning","papers":2}],"tasks_shown":20,"n_tasks":55,"usage_by_year":[{"year":"2015","papers":1},{"year":"2016","papers":1},{"year":"2017","papers":5},{"year":"2018","papers":5},{"year":"2019","papers":12},{"year":"2020","papers":12},{"year":"2021","papers":10},{"year":"2022","papers":7},{"year":"2023","papers":14},{"year":"2024","papers":9},{"year":"2025","papers":5}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/trpo"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}