Methods › Reinforcement Learning › Policy Gradient Methods › TRPO
Trust Region Policy Optimization
TRPO
Introduced by John Schulman et al. in Trust Region Policy Optimization
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Trust Region Policy Optimization, or TRPO, is a policy gradient method in reinforcement learning that avoids parameter updates that change the policy too much with a KL divergence constraint on the size of the policy update at each iteration.
Take the case of off-policy reinforcement learning, where the policy β for collecting trajectories on rollout workers is different from the policy π to optimize for. The objective function in an off-policy model measures the total advantage over the state visitation distribution and actions, while the mismatch between the training data distribution and the true policy state distribution is compensated with an importance sampling estimator:
J(θ) = ∑_(s∈S)p^(π_(θ_(old)))∑_(a∈𝒜)(π_θ(a|s)Â_(θ_(old))(s, a))
J(θ) = ∑_(s∈S)p^(π_(θ_(old)))∑_(a∈𝒜)(β(a|s)(π_θ(a|s))/(β(a|s))Â_(θ_(old))(s, a))
J(θ) = 𝔼_(s∼p^(π_(θ_(old))), a∼β) ((π_θ(a|s))/(β(a|s))Â_(θ_(old))(s, a))
When training on policy, theoretically the policy for collecting data is same as the policy that we want to optimize. However, when rollout workers and optimizers are running in parallel asynchronously, the behavior policy can get stale. TRPO considers this subtle difference: It labels the behavior policy as π_(θ_(old))(a|s) and thus the objective function becomes:
J(θ) = 𝔼_(s∼p^(π_(θ_(old))), a∼π_(θ_(old))) ((π_θ(a|s))/(π_(θ_(old))(a|s))Â_(θ_(old))(s, a))
TRPO aims to maximize the objective function J(θ) subject to a trust region constraint which enforces the distance between old and new policies measured by KL-divergence to be small enough, within a parameter δ:
𝔼_(s∼p^(π_(θ_(old)))) [D_(KL)(π_(θ_(old))(.|s)||π_θ(.|s))] ≤δ
In Syntology about another paper, arXiv:2606.17664: repository known, no verified implementation (declared_code; repositories: 1 listed of 1 in the graph, strongest link: listed only). About another paper: this answer is about arXiv:2606.17664, while the archive credits arXiv:1502.05477.
Code Syntology holds under other names, as the lookup lists them (code_nearby); not a complete list, and none of them is this method.
- Optimal Transport Trust Region Policy Optimization (OT-TRPO) · shares_name_tokens (shares name tokens with this name; a different method) · level V2 · retrieve with
get_reference_implementation(name='Optimal Transport Trust Region Policy Optimization (OT-TRPO)')
Origin paper of this answer, arXiv:2606.17664 (its page on this site · its page on Syntology): Syntology's run record for its harvested code, 4 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 5 harvested samples) · 5 pointer-only (licence). The instrument figure counts failures of Syntology's instrument, not of the code. Ran means executed on a synthesized input, not that the code is correct.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
What each level means, quoted from Syntology's server card: V0 “imports”; V1 “executes against a synthesized fixture and returns a value of the recorded type”; V2 “two implementations GENERATED from the paper's signature and spec agree numerically (1e-6) on fixture inputs” “neither of the two ever read the authors' own code, so V2 measures how unambiguously the paper specifies the method, not fidelity to the authors' implementation (measured 2026-09-11, commit 75eaa77: cross_checked_by is a second LLM on 2,628 of 2,831 served samples)”; V3 “paper-derived property tests pass on both of those same two generations.” The card adds: “No rung on this ladder compares the served code against the authors' implementation, so treat every level as evidence about runnability and internal agreement, not about matching the paper's results.” A level quoted in two parts is one definition the card writes as two clauses joined by a double hyphen, each clause quoted on its own (how these levels are quoted).
Papers archive 2025-07-28
archive papers tagged: 81 · with a code link: 36 · where Syntology ran a sample: 10 (9 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (10 of 81 tagged: 9 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 81 of 81, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
StaQ it! Growing neural networks for Policy Mirror Descent 16 Jun 2025 · 0 repositories · arXiv:2506.13862
-
Finite-Sample Convergence Bounds for Trust Region Policy Optimization in Mean-Field Games 28 May 2025 · 0 repositories · arXiv:2505.22781
-
Improving Value Estimation Critically Enhances Vanilla Policy Gradient 25 May 2025 · 1 repository · arXiv:2505.19247
-
Energy Efficient RSMA-Based LEO Satellite Communications Assisted by UAV-Mounted BD-Active RIS: A DRL Approach 7 May 2025 · 0 repositories · arXiv:2505.04148
-
Deep Reinforcement Learning-Based User Association in Hybrid LiFi/WiFi Indoor Networks 3 Mar 2025 · 0 repositories · arXiv:2503.01803
-
Fast Convergence of Softmax Policy Mirror Ascent 18 Nov 2024 · 0 repositories · arXiv:2411.12042
-
Dynamics of Resource Allocation in O-RANs: An In-depth Exploration of On-Policy and Off-Policy Deep Reinforcement Learning for Real-Time Applications 17 Nov 2024 · 0 repositories · arXiv:2412.01839
-
Embedding Safety into RL: A New Take on Trust Region Methods 5 Nov 2024 · 0 repositories · arXiv:2411.02957
-
Linear Function Approximation as a Computationally Efficient Method to Solve Classical Reinforcement Learning Challenges 27 May 2024 · 0 repositories · arXiv:2405.20350
-
Matrix Low-Rank Trust Region Policy Optimization 27 May 2024 · 1 repository · arXiv:2405.17625
-
Joint Physical-Digital Facial Attack Detection Via Simulating Spoofing Clues 12 Apr 2024 · 3 repositories · arXiv:2404.08450
-
Policy Mirror Descent with Lookahead 21 Mar 2024 · 1 repository · arXiv:2403.14156
-
Convergence for Natural Policy Gradient on Infinite-State Queueing MDPs 7 Feb 2024 · 0 repositories · arXiv:2402.05274
-
Simple Policy Optimization 29 Jan 2024 · 1 repository · arXiv:2401.16025
-
Clipped-Objective Policy Gradients for Pessimistic Policy Optimization 10 Nov 2023 · 1 repository · arXiv:2311.05846
-
Dropout Strategy in Reinforcement Learning: Limiting the Surrogate Objective Variance in Policy Optimization Methods 31 Oct 2023 · 0 repositories · arXiv:2310.20380
-
Distributional Soft Actor-Critic with Three Refinements 9 Oct 2023 · 2 repositories · arXiv:2310.05858Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 0 violated, 3 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 5 harvested samples) · 5 pointer-only (licence)
-
General Munchausen Reinforcement Learning with Tsallis Kullback-Leibler Divergence 21 Sep 2023 · 0 repositories
-
Distributional Estimation of Data Uncertainty for Surveillance Face Anti-spoofing 18 Sep 2023 · 0 repositories · arXiv:2309.09485
-
ContainerGym: A Real-World Reinforcement Learning Benchmark for Resource Allocation 6 Jul 2023 · 1 repository · arXiv:2307.02991
-
Safe, Efficient, Comfort, and Energy-saving Automated Driving through Roundabout Based on Deep Reinforcement Learning 20 Jun 2023 · 0 repositories · arXiv:2306.11465
-
Decision-Aware Actor-Critic with Function Approximation and Theoretical Guarantees 24 May 2023 · 1 repository · arXiv:2305.15249Syntology official (archive's flag): 1 ran · 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified; the one sample that ran constructed an object rather than computing a result (of 2 harvested samples) · 2 pointer-only (licence)
-
Trust-Region-Free Policy Optimization for Stochastic Policies 15 Feb 2023 · 0 repositories · arXiv:2302.07985
-
Sample Dropout: A Simple yet Effective Variance Reduction Technique in Deep Policy Optimization 5 Feb 2023 · 1 repository · arXiv:2302.02299Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 0 violated, 3 with no contract checked; 2 where Syntology's instrument failed) · 6 unverified (of 12 harvested samples) · 12 pointer-only (licence)
-
Learning, Fast and Slow: A Goal-Directed Memory-Based Approach for Dynamic Environments 31 Jan 2023 · 1 repository · arXiv:2301.13758
-
A Novel Framework for Policy Mirror Descent with General Parameterization and Linear Convergence 30 Jan 2023 · 1 repository · arXiv:2301.13139
-
Generalized Munchausen Reinforcement Learning using Tsallis KL Divergence 27 Jan 2023 · 0 repositories · arXiv:2301.11476
-
PDVN: A Patch-based Dual-view Network for Face Liveness Detection using Light Field Focal Stack 17 Jan 2023 · 0 repositories
-
Relative Sparsity for Medical Decision Problems 29 Nov 2022 · 1 repository · arXiv:2211.16566
-
Normality-Guided Distributional Reinforcement Learning for Continuous Control 28 Aug 2022 · 0 repositories · arXiv:2208.13125
-
Entropy Augmented Reinforcement Learning 19 Aug 2022 · 0 repositories · arXiv:2208.09322
-
Asynchronous Curriculum Experience Replay: A Deep Reinforcement Learning Approach for UAV Autonomous Motion Control in Unknown Dynamic Environments 4 Jul 2022 · 0 repositories · arXiv:2207.01251
-
Truly Deterministic Policy Optimization 30 May 2022 · 1 repository · arXiv:2205.15379
-
Is Word Error Rate a good evaluation metric for Speech Recognition in Indic Languages? 30 Mar 2022 · 0 repositories · arXiv:2203.16601
-
Mirror Learning: A Unifying Framework of Policy Optimisation 7 Jan 2022 · 1 repository · arXiv:2201.02373
-
Learning Reward Machines: A Study in Partially Observable Reinforcement Learning 17 Dec 2021 · 0 repositories · arXiv:2112.09477
-
EnTRPO: Trust Region Policy Optimization Method with Entropy Regularization 26 Oct 2021 · 0 repositories · arXiv:2110.13373
-
A general class of surrogate functions for stable and efficient reinforcement learning 12 Aug 2021 · 1 repository · arXiv:2108.05828
-
Refined Policy Improvement Bounds for MDPs 16 Jul 2021 · 0 repositories · arXiv:2107.08068
-
Curious Explorer: a provable exploration strategy in Policy Learning 29 Jun 2021 · 0 repositories · arXiv:2106.15503
-
On-Policy Deep Reinforcement Learning for the Average-Reward Criterion 14 Jun 2021 · 0 repositories · arXiv:2106.07329
-
On the Linear convergence of Natural Policy Gradient Algorithm 4 May 2021 · 0 repositories · arXiv:2105.01424
-
A-DeepPixBis: Attentional Angular Margin for Face Anti-Spoofing 1 Mar 2021 · 0 repositories
-
Optimization Issues in KL-Constrained Approximate Policy Iteration 11 Feb 2021 · 0 repositories · arXiv:2102.06234
-
A review of motion planning algorithms for intelligent robotics 4 Feb 2021 · 0 repositories · arXiv:2102.02376
-
Trust Region Policy Optimization Regularized via PID Control 18 Dec 2020 · 0 repositories
-
Tonic: A Deep Reinforcement Learning Library for Fast Prototyping and Benchmarking 15 Nov 2020 · 1 repository · arXiv:2011.07537Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified (of 4 harvested samples)
-
Control with adaptive Q-learning 3 Nov 2020 · 1 repository · arXiv:2011.02141
-
Multi-Agent Trust Region Policy Optimization 15 Oct 2020 · 1 repository · arXiv:2010.07916Syntology 16 ran (of which 0 constructed an object rather than computing a result; 14 with no instrument failure: 0 honoured, 0 violated, 14 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified (of 18 harvested samples) · 7 pointer-only (licence)
-
Faded-Experience Trust Region Policy Optimization for Model-Free Power Allocation in Interference Channel 4 Aug 2020 · 0 repositories · arXiv:2008.01705
-
Lagrangian Duality in Reinforcement Learning 20 Jul 2020 · 0 repositories · arXiv:2007.09998
-
Optimistic Distributionally Robust Policy Optimization 14 Jun 2020 · 1 repository · arXiv:2006.07815Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 5 harvested samples)
-
Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO 25 May 2020 · 3 repositories · arXiv:2005.12729Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)
-
Mirror Descent Policy Optimization 20 May 2020 · 1 repository · arXiv:2005.09814
-
Implementation Matters in Deep RL: A Case Study on PPO and TRPO 1 May 2020 · 2 repositories
-
First Order Constrained Optimization in Policy Space 16 Feb 2020 · 2 repositories · arXiv:2002.06506
-
Analyzing Policy Distillation on Multi-Task Learning and Meta-Reinforcement Learning in Meta-World 8 Feb 2020 · 0 repositories
-
Risk-Averse Trust Region Optimization for Reward-Volatility Reduction 6 Dec 2019 · 0 repositories · arXiv:1912.03193
-
Learning Reward Machines for Partially Observable Reinforcement Learning 1 Dec 2019 · 1 repository
-
Neural Trust Region/Proximal Policy Optimization Attains Globally Optimal Policy 1 Dec 2019 · 0 repositories
-
Multi-step Greedy Reinforcement Learning Algorithms 7 Oct 2019 · 0 repositories · arXiv:1910.02919
-
Revisit Policy Optimization in Matrix Form 19 Sep 2019 · 0 repositories · arXiv:1909.09186
-
Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPs 6 Sep 2019 · 0 repositories · arXiv:1909.02769
-
Hindsight Trust Region Policy Optimization 29 Jul 2019 · 1 repository · arXiv:1907.12439
-
Neural Proximal/Trust Region Policy Optimization Attains Globally Optimal Policy 25 Jun 2019 · 0 repositories · arXiv:1906.10306
-
Policy Optimization via Stochastic Recursive Gradient Algorithm 1 May 2019 · 0 repositories
-
SUPERVISED POLICY UPDATE 1 May 2019 · 1 repository
-
Discretizing Continuous Action Space for On-Policy Optimization 29 Jan 2019 · 2 repositories · arXiv:1901.10500
-
On-Policy Trust Region Policy Optimisation with Replay Buffers 18 Jan 2019 · 2 repositories · arXiv:1901.06212
-
Multi-objective Model-based Policy Search for Data-efficient Learning with Sparse Rewards 25 Jun 2018 · 1 repository · arXiv:1806.09351
-
Supervised Policy Update for Deep Reinforcement Learning 29 May 2018 · 1 repository · arXiv:1805.11706
-
Variational Inference for Policy Gradient 21 Feb 2018 · 0 repositories · arXiv:1802.07833
-
Sample Efficient Deep Reinforcement Learning for Dialogue Systems with Large Action Spaces 11 Feb 2018 · 0 repositories · arXiv:1802.03753
-
Pretraining Deep Actor-Critic Reinforcement Learning Algorithms With Expert Demonstrations 31 Jan 2018 · 0 repositories · arXiv:1801.10459
-
Multi-task Learning with Gradient Guided Policy Specialization 23 Sep 2017 · 0 repositories · arXiv:1709.07979
-
Trust-PCL: An Off-Policy Trust Region Method for Continuous Control 6 Jul 2017 · 1 repository · arXiv:1707.01891
-
Parameter Space Noise for Exploration 6 Jun 2017 · 10 repositories · arXiv:1706.01905Syntology 4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 1 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 1 unverified (of 5 harvested samples) · 5 pointer-only (licence)
-
A unified view of entropy-regularized Markov decision processes 22 May 2017 · 0 repositories · arXiv:1705.07798
-
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks 9 Mar 2017 · 85 repositories · arXiv:1703.03400Syntology official (archive's flag): 1 ran · 103 ran (of which 35 constructed an object rather than computing a result; 72 with no instrument failure: 6 honoured, 1 violated, 65 with no contract checked; 31 where Syntology's instrument failed) · 51 unverified (of 154 harvested samples) · 57 pointer-only (licence)
-
Sample Efficient Actor-Critic with Experience Replay 3 Nov 2016 · 7 repositories · arXiv:1611.01224Syntology 0 ran · 1 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
Trust Region Policy Optimization 19 Feb 2015 · 24 repositories · arXiv:1502.05477Syntology 7 ran (of which 2 constructed an object rather than computing a result; 6 with no instrument failure: 2 honoured, 0 violated, 4 with no contract checked; 1 where Syntology's instrument failed) · 10 unverified (of 17 harvested samples) · 7 pointer-only (licence)
Tasks archive 2025-07-28
20 shown of 55 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections