</> Code with Papers by Syntology

Methods › Reinforcement Learning › Policy Gradient Methods › TRPO

Trust Region Policy Optimization

TRPO

81 papers tagged archive 2025-07-28

Introduced by John Schulman et al. in Trust Region Policy Optimization

archive 2025-07-28 Description, source and code snippet are the archive's method entry.

Trust Region Policy Optimization, or TRPO, is a policy gradient method in reinforcement learning that avoids parameter updates that change the policy too much with a KL divergence constraint on the size of the policy update at each iteration.

Take the case of off-policy reinforcement learning, where the policy β for collecting trajectories on rollout workers is different from the policy π to optimize for. The objective function in an off-policy model measures the total advantage over the state visitation distribution and actions, while the mismatch between the training data distribution and the true policy state distribution is compensated with an importance sampling estimator:

J(θ) = ∑_(s∈S)p^(π_(θ_(old)))∑_(a∈𝒜)(π_θ(a|s)Â_(θ_(old))(s, a))

J(θ) = ∑_(s∈S)p^(π_(θ_(old)))∑_(a∈𝒜)(β(a|s)(π_θ(a|s))/(β(a|s))Â_(θ_(old))(s, a))

J(θ) = 𝔼_(s∼p^(π_(θ_(old))), a∼β) ((π_θ(a|s))/(β(a|s))Â_(θ_(old))(s, a))

When training on policy, theoretically the policy for collecting data is same as the policy that we want to optimize. However, when rollout workers and optimizers are running in parallel asynchronously, the behavior policy can get stale. TRPO considers this subtle difference: It labels the behavior policy as π_(θ_(old))(a|s) and thus the objective function becomes:

J(θ) = 𝔼_(s∼p^(π_(θ_(old))), a∼π_(θ_(old))) ((π_θ(a|s))/(π_(θ_(old))(a|s))Â_(θ_(old))(s, a))

TRPO aims to maximize the objective function J(θ) subject to a trust region constraint which enforces the distance between old and new policies measured by KL-divergence to be small enough, within a parameter δ:

𝔼_(s∼p^(π_(θ_(old)))) [D_(KL)(π_(θ_(old))(.|s)||π_θ(.|s))] ≤δ

PaperSource

Papers archive 2025-07-28

30 shown of 81, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.

Tasks archive 2025-07-28

20 shown of 55 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.

TaskPapers
Reinforcement Learning (RL)46
Reinforcement Learning32
reinforcement-learning30
Deep Reinforcement Learning23
Continuous Control11
Policy Gradient Methods9
MuJoCo8
continuous-control8
Decision Making5
Atari Games3
Face Anti-Spoofing3
Face Recognition3
Multi-Task Learning3
Benchmarking2
Meta-Learning2
Model-based Reinforcement Learning2
OpenAI Gym2
Partially Observable Reinforcement Learning2
Problem Decomposition2
Q-Learning2

Usage over time archive 2025-07-28

Papers per year tagged with TRPO: 2015 to 2025, peak 14 14 0 2015: 1 paper 2015 2016: 1 paper 2016 2017: 5 papers 2017 2018: 5 papers 2018 2019: 12 papers 2019 2020: 12 papers 2020 2021: 10 papers 2021 2022: 7 papers 2022 2023: 14 papers 2023 2024: 9 papers 2024 2025: 5 papers 2025
Papers per year the archive tags with this method, by the paper's archive date (81 dated). Bars are counts, not a trend claim.

Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).

Categories archive 2025-07-28

Policy Gradient Methods

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections