Methods › Reinforcement Learning › Policy Gradient Methods › PPO
Proximal Policy Optimization
PPO
Introduced by John Schulman et al. in Proximal Policy Optimization Algorithms
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Proximal Policy Optimization, or PPO, is a policy gradient method for reinforcement learning. The motivation was to have an algorithm with the data efficiency and reliable performance of TRPO, while using only first-order optimization.
Let rₜ(θ) denote the probability ratio rₜ(θ) = (π_θ(aₜ|sₜ))/(π_(θ_(old))(aₜ|sₜ)), so r(θ_(old)) = 1. TRPO maximizes a “surrogate” objective:
L^(CPI)(θ) = 𝔼̂ₜ[(π_θ(aₜ|sₜ))/(π_(θ_(old))(aₜ|sₜ)))Âₜ] = 𝔼̂ₜ[rₜ(θ)Âₜ]
Where CPI refers to a conservative policy iteration. Without a constraint, maximization of L^(CPI) would lead to an excessively large policy update; hence, we PPO modifies the objective, to penalize changes to the policy that move rₜ(θ) away from 1:
J^(CLIP)(θ) = 𝔼̂ₜ[min(rₜ(θ)Âₜ, clip(rₜ(θ), 1-ϵ, 1+ϵ)Âₜ)]
where ϵ is a hyperparameter, say, ϵ= 0.2. The motivation for this objective is as follows. The first term inside the min is L^(CPI). The second term, clip(rₜ(θ), 1-ϵ, 1+ϵ)Âₜ modifies the surrogate objective by clipping the probability ratio, which removes the incentive for moving rₜ outside of the interval [1 − ϵ, 1 + ϵ]. Finally, we take the minimum of the clipped and unclipped objective, so the final objective is a lower bound (i.e., a pessimistic bound) on the unclipped objective. With this scheme, we only ignore the change in probability ratio when it would make the objective improve, and we include it when it makes the objective worse.
One detail to note is that when we apply PPO for a network where we have shared parameters for actor and critic functions, we typically add to the objective function an error term on value estimation and an entropy term to encourage exploration.
In Syntology repository known, no verified implementation (declared_code; repositories: 4 listed of 187 in the graph, strongest link: mentioned on GitHub). The lookup matched a different name in Syntology's graph, 'Proximal Policy Optimization (PPO)' (resolution: alias); this answer is about that name. This answer is about arXiv:1707.06347, the paper the archive credits with this method.
Origin paper of this answer, arXiv:1707.06347 (its page on this site · its page on Syntology): Syntology's run record for its harvested code, 99 ran (of which 53 constructed an object rather than computing a result; 71 with no instrument failure: 7 honoured, 2 violated, 62 with no contract checked; 28 where Syntology's instrument failed) · 77 unverified (of 176 harvested samples) · 94 pointer-only (licence). The instrument figure counts failures of Syntology's instrument, not of the code. Ran means executed on a synthesized input, not that the code is correct.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 949 · with a code link: 397 · where Syntology ran a sample: 139 (112 with a run with no instrument failure, 27 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (139 of 949 tagged: 112 with a run with no instrument failure, 27 where every run was a failure of Syntology's instrument)
Page 1 of 10: papers 1 to 100 of 949, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
AirLLM: Diffusion Policy-based Adaptive LoRA for Remote Fine-Tuning of LLM over the Air 15 Jul 2025 · 0 repositories · arXiv:2507.11515
-
Step-wise Policy for Rare-tool Knowledge (SPaRK): Offline RL that Drives Diverse Tool Use in LLMs 15 Jul 2025 · 1 repository · arXiv:2507.11371
-
Scene-Aware Conversational ADAS with Generative AI for Real-Time Driver Assistance 14 Jul 2025 · 0 repositories · arXiv:2507.10500
-
Meta-Reinforcement Learning for Fast and Data-Efficient Spectrum Allocation in Dynamic Wireless Networks 13 Jul 2025 · 0 repositories · arXiv:2507.10619
-
Deep Reinforcement Learning with Gradient Eligibility Traces 12 Jul 2025 · 1 repository · arXiv:2507.09087
-
Geo-ORBIT: A Federated Digital Twin Framework for Scene-Adaptive Lane Geometry Detection 11 Jul 2025 · 1 repository · arXiv:2507.08743
-
LeAD: The LLM Enhanced Planning System Converged with End-to-end Autonomous Driving 8 Jul 2025 · 0 repositories · arXiv:2507.05754
-
Model-free Optical Processors using In Situ Reinforcement Learning with Proximal Policy Optimization 8 Jul 2025 · 0 repositories · arXiv:2507.05583
-
2048: Reinforcement Learning in a Delayed Reward Environment 7 Jul 2025 · 0 repositories · arXiv:2507.05465
-
YOLO-APD: Enhancing YOLOv8 for Robust Pedestrian Detection on Complex Road Geometries 7 Jul 2025 · 0 repositories · arXiv:2507.05376
-
LLM-based Realistic Safety-Critical Driving Video Generation 2 Jul 2025 · 0 repositories · arXiv:2507.01264
-
BIDA: A Bi-level Interaction Decision-making Algorithm for Autonomous Vehicles in Dynamic Traffic Scenarios 19 Jun 2025 · 0 repositories · arXiv:2506.16546
-
Multi-Agent Reinforcement Learning for Autonomous Multi-Satellite Earth Observation: A Realistic Case Study 18 Jun 2025 · 0 repositories · arXiv:2506.15207
-
Algorithmic Approaches to Enhance Safety in Autonomous Vehicles: Minimizing Lane Changes and Merging 17 Jun 2025 · 0 repositories · arXiv:2506.15026
-
Light Aircraft Game : Basic Implementation and training results analysis 17 Jun 2025 · 1 repository · arXiv:2506.14164
-
TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization 17 Jun 2025 · 1 repository · arXiv:2506.14574Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models 16 Jun 2025 · 0 repositories · arXiv:2506.13923
-
AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning 16 Jun 2025 · 1 repository · arXiv:2506.13757Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
How Real is CARLAs Dynamic Vision Sensor? A Study on the Sim-to-Real Gap in Traffic Object Detection 16 Jun 2025 · 0 repositories · arXiv:2506.13722
-
Ego-centric Learning of Communicative World Models for Autonomous Driving 9 Jun 2025 · 0 repositories · arXiv:2506.08149
-
Autonomous Vehicle Lateral Control Using Deep Reinforcement Learning with MPC-PID Demonstration 4 Jun 2025 · 0 repositories · arXiv:2506.04040
-
PPO in the Fisher-Rao geometry 4 Jun 2025 · 0 repositories · arXiv:2506.03757
-
RedRFT: A Light-Weight Benchmark for Reinforcement Fine-Tuning-Based Red Teaming 4 Jun 2025 · 1 repository · arXiv:2506.04302
-
Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective 3 Jun 2025 · 0 repositories · arXiv:2506.02553
-
Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening 3 Jun 2025 · 0 repositories · arXiv:2506.02355
-
The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning 2 Jun 2025 · 1 repository · arXiv:2506.01347Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)
-
DriveMind: A Dual-VLM based Reinforcement Learning Framework for Autonomous Driving 1 Jun 2025 · 0 repositories · arXiv:2506.00819
-
Language-Guided Multi-Agent Learning in Simulations: A Unified Framework and Evaluation 1 Jun 2025 · 0 repositories · arXiv:2506.04251
-
Using Diffusion Ensembles to Estimate Uncertainty for End-to-End Autonomous Driving 31 May 2025 · 0 repositories · arXiv:2506.00560
-
AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning 30 May 2025 · 1 repository · arXiv:2505.24298Syntology official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified (of 13 harvested samples)
-
Composite Reward Design in PPO-Driven Adaptive Filtering 29 May 2025 · 1 repository · arXiv:2506.06323
-
Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models 29 May 2025 · 1 repository · arXiv:2505.23564Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified (of 9 harvested samples)
-
R1-Code-Interpreter: Training LLMs to Reason with Code via Supervised and Reinforcement Learning 27 May 2025 · 1 repository · arXiv:2505.21668
-
What Can RL Bring to VLA Generalization? An Empirical Study 26 May 2025 · 0 repositories · arXiv:2505.19789
-
Improving Value Estimation Critically Enhances Vanilla Policy Gradient 25 May 2025 · 1 repository · arXiv:2505.19247
-
GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learning 24 May 2025 · 0 repositories · arXiv:2505.18763
-
A Robust PPO-optimized Tabular Transformer Framework for Intrusion Detection in Industrial IoT Systems 23 May 2025 · 1 repository · arXiv:2505.18234
-
ShIOEnv: A CLI Behavior-Capturing Environment Enabling Grammar-Guided Command Synthesis for Dataset Curation 23 May 2025 · 1 repository · arXiv:2505.18374
-
The Cell Must Go On: Agar.io for Continual Reinforcement Learning 23 May 2025 · 1 repository · arXiv:2505.18347
-
Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving (in CARLA v2) 22 May 2025 · 0 repositories · arXiv:2505.16394
-
AM-PPO: (Advantage) Alpha-Modulation with Proximal Policy Optimization 21 May 2025 · 0 repositories · arXiv:2505.15514
-
HCRMP: A LLM-Hinted Contextual Reinforcement Learning Framework for Autonomous Driving 21 May 2025 · 0 repositories · arXiv:2505.15793
-
iPad: Iterative Proposal-centric End-to-End Autonomous Driving 21 May 2025 · 1 repository · arXiv:2505.15111
-
Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning 21 May 2025 · 0 repositories · arXiv:2505.15311Syntology 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)
-
Interpretable Reinforcement Learning for Load Balancing using Kolmogorov-Arnold Networks 20 May 2025 · 0 repositories · arXiv:2505.14459
-
KIPPO: Koopman-Inspired Proximal Policy Optimization 20 May 2025 · 0 repositories · arXiv:2505.14566
-
Power Allocation for Delay Optimization in Device-to-Device Networks: A Graph Reinforcement Learning Approach 19 May 2025 · 0 repositories · arXiv:2505.12902
-
Bi-Level Policy Optimization with Nyström Hypergradients 16 May 2025 · 0 repositories · arXiv:2505.11714
-
Reinforcement Learning Finetunes Small Subnetworks in Large Language Models 16 May 2025 · 0 repositories · arXiv:2505.11711Syntology 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
Fine-tuning Diffusion Policies with Backpropagation Through Diffusion Timesteps 15 May 2025 · 0 repositories · arXiv:2505.10482
-
Monte Carlo Beam Search for Actor-Critic Reinforcement Learning in Continuous Control 13 May 2025 · 0 repositories · arXiv:2505.09029
-
Object detection in adverse weather conditions for autonomous vehicles using Instruct Pix2Pix 13 May 2025 · 0 repositories · arXiv:2505.08228
-
DialogueReason: Rule-Based RL Sparks Dialogue Reasoning in LLMs 11 May 2025 · 0 repositories · arXiv:2505.07049
-
A critical assessment of reinforcement learning methods for microswimmer navigation in complex flows 8 May 2025 · 1 repository · arXiv:2505.05525
-
Multi-Objective Reinforcement Learning for Adaptive Personalized Autonomous Driving 8 May 2025 · 0 repositories · arXiv:2505.05223
-
Reinforcement Learning for Game-Theoretic Resource Allocation on Graphs 8 May 2025 · 0 repositories · arXiv:2505.06319
-
The City that Never Settles: Simulation-based LiDAR Dataset for Long-Term Place Recognition Under Extreme Structural Changes 8 May 2025 · 1 repository · arXiv:2505.05076
-
X-Driver: Explainable Autonomous Driving with Vision-Language Models 8 May 2025 · 0 repositories · arXiv:2505.05098
-
Putting the Value Back in RL: Better Test-Time Scaling by Unifying LLM Reasoners With Verifiers 7 May 2025 · 0 repositories · arXiv:2505.04842
-
Interpretable Learning Dynamics in Unsupervised Reinforcement Learning 6 May 2025 · 0 repositories · arXiv:2505.06279
-
HyperController: A Hyperparameter Controller for Fast and Stable Training of Reinforcement Learning Neural Networks 27 Apr 2025 · 1 repository · arXiv:2504.19382
-
Federated Learning-based Semantic Segmentation for Lane and Object Detection in Autonomous Driving 26 Apr 2025 · 0 repositories · arXiv:2504.18939
-
Hierarchical Reinforcement Learning in Multi-Goal Spatial Navigation with Autonomous Mobile Robots 26 Apr 2025 · 0 repositories · arXiv:2504.18794
-
CaRL: Learning Scalable Planning Policies with Simple Rewards 24 Apr 2025 · 1 repository · arXiv:2504.17838Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 1 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
Training Large Language Models to Reason via EM Policy Gradient 24 Apr 2025 · 0 repositories · arXiv:2504.18587
-
AlphaGrad: Non-Linear Gradient Normalization Optimizer 22 Apr 2025 · 0 repositories · arXiv:2504.16020
-
LangCoop: Collaborative Driving with Language 18 Apr 2025 · 1 repository · arXiv:2504.13406
-
Evaluating the Diversity and Quality of LLM Generated Content 16 Apr 2025 · 0 repositories · arXiv:2504.12522
-
A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce 15 Apr 2025 · 1 repository · arXiv:2504.11343Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
Measures of Variability for Risk-averse Policy Gradient 15 Apr 2025 · 0 repositories · arXiv:2504.11412
-
Iterative Self-Training for Code Generation via Reinforced Re-Ranking 13 Apr 2025 · 0 repositories · arXiv:2504.09643
-
A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems 12 Apr 2025 · 0 repositories · arXiv:2504.09037
-
A Reinforcement Learning Method for Environments with Stochastic Variables: Post-Decision Proximal Policy Optimization with Dual Critic Networks 7 Apr 2025 · 1 repository · arXiv:2504.05150
-
Concise Reasoning via Reinforcement Learning 7 Apr 2025 · 1 repository · arXiv:2504.05185Syntology official (archive's flag): 11 ran · 12 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 1 honoured, 5 violated, 0 with no contract checked; 6 where Syntology's instrument failed) · 2 unverified (of 14 harvested samples) · 13 pointer-only (licence)
-
Deep Reinforcement Learning Algorithms for Option Hedging 7 Apr 2025 · 1 repository · arXiv:2504.05521
-
Resource-Efficient Beam Prediction in mmWave Communications with Multimodal Realistic Simulation Framework 7 Apr 2025 · 0 repositories · arXiv:2504.05187
-
CORTEX-AVD: A Framework for CORner Case Testing and EXploration in Autonomous Vehicle Development 4 Apr 2025 · 0 repositories · arXiv:2504.03989
-
End-to-End Driving with Online Trajectory Evaluation via BEV World Model 2 Apr 2025 · 1 repository · arXiv:2504.01941Syntology official (archive's flag): 7 ran · 7 ran (of which 7 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified; every one of the 7 samples that ran constructed an object rather than computing a result (of 11 harvested samples)
-
Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model 31 Mar 2025 · 1 repository · arXiv:2503.24290Syntology 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified (of 4 harvested samples)
-
UniOcc: A Unified Benchmark for Occupancy Forecasting and Prediction in Autonomous Driving 31 Mar 2025 · 1 repository · arXiv:2503.24381Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (of 6 harvested samples)
-
On the Mistaken Assumption of Interchangeable Deep Reinforcement Learning Implementations 28 Mar 2025 · 0 repositories · arXiv:2503.22575
-
Knowledge Graphs as World Models for Semantic Material-Aware Obstacle Handling in Autonomous Vehicles 27 Mar 2025 · 0 repositories · arXiv:2503.21232
-
GAPO: Learning Preferential Prompt through Generative Adversarial Policy Optimization 26 Mar 2025 · 1 repository · arXiv:2503.20194
-
Evolutionary Policy Optimization 24 Mar 2025 · 0 repositories · arXiv:2503.19037
-
Curriculum RL meets Monte Carlo Planning: Optimization of a Real World Container Management Problem 21 Mar 2025 · 1 repository · arXiv:2503.17194
-
Optimizing 2D+1 Packing in Constrained Environments Using Deep Reinforcement Learning 21 Mar 2025 · 0 repositories · arXiv:2503.17573
-
RESFL: An Uncertainty-Aware Framework for Responsible Federated Learning by Balancing Privacy, Fairness and Utility in Autonomous Vehicles 20 Mar 2025 · 0 repositories · arXiv:2503.16251
-
Good Actions Succeed, Bad Actions Generalize: A Case Study on Why RL Generalizes Better 19 Mar 2025 · 0 repositories · arXiv:2503.15693
-
Reinforcement learning-based motion imitation for physiologically plausible musculoskeletal motor control 18 Mar 2025 · 1 repository · arXiv:2503.14637
-
DiffAD: A Unified Diffusion Modeling Approach for Autonomous Driving 15 Mar 2025 · 0 repositories · arXiv:2503.12170
-
Exploring Competitive and Collusive Behaviors in Algorithmic Pricing with Deep Reinforcement Learning 14 Mar 2025 · 0 repositories · arXiv:2503.11270
-
An Real-Sim-Real (RSR) Loop Framework for Generalizable Robotic Policy Transfer with Differentiable Simulation 13 Mar 2025 · 1 repository · arXiv:2503.10118
-
Edge AI-Powered Real-Time Decision-Making for Autonomous Vehicles in Adverse Weather Conditions 12 Mar 2025 · 0 repositories · arXiv:2503.09638
-
ManeuverGPT Agentic Control for Safe Autonomous Stunt Maneuvers 12 Mar 2025 · 1 repository · arXiv:2503.09035
-
SimLingo: Vision-Only Closed-Loop Autonomous Driving with Language-Action Alignment 12 Mar 2025 · 1 repository · arXiv:2503.09594Syntology 11 ran (of which 8 constructed an object rather than computing a result; 8 with no instrument failure: 0 honoured, 0 violated, 8 with no contract checked; 3 where Syntology's instrument failed) · 3 unverified (of 14 harvested samples)
-
JiSAM: Alleviate Labeling Burden and Corner Case Problems in Autonomous Driving via Minimal Real-World Data 11 Mar 2025 · 0 repositories · arXiv:2503.08422
-
Task-Oriented Co-Design of Communication, Computing, and Control for Edge-Enabled Industrial Cyber-Physical Systems 11 Mar 2025 · 0 repositories · arXiv:2503.08661
-
Combating Partial Perception Deficit in Autonomous Driving with Multimodal LLM Commonsense 10 Mar 2025 · 0 repositories · arXiv:2503.07020
-
Using a single actor to output personalized policy for different intersections 10 Mar 2025 · 0 repositories · arXiv:2503.07678
-
GFlowVLM: Enhancing Multi-step Reasoning in Vision-Language Models with Generative Flow Networks 9 Mar 2025 · 0 repositories · arXiv:2503.06514
Tasks archive 2025-07-28
20 shown of 357 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections