{"url":"/method/ppo","slug":"ppo","name":"PPO","full_name":"Proximal Policy Optimization","full_name_withheld":false,"description_markdown":"**Proximal Policy Optimization**, or **PPO**, is a policy gradient method for reinforcement learning. The motivation was to have an algorithm with the data efficiency and reliable performance of [TRPO](https://paperswithcode.com/method/trpo), while using only first-order optimization. \r\n\r\nLet $r\\_{t}\\left(\\theta\\right)$ denote the probability ratio $r\\_{t}\\left(\\theta\\right) = \\frac{\\pi\\_{\\theta}\\left(a\\_{t}\\mid{s\\_{t}}\\right)}{\\pi\\_{\\theta\\_{old}}\\left(a\\_{t}\\mid{s\\_{t}}\\right)}$, so $r\\left(\\theta\\_{old}\\right) = 1$. TRPO maximizes a “surrogate” objective:\r\n\r\n$$ L^{\\text{CPI}}\\left({\\theta}\\right) = \\hat{\\mathbb{E}}\\_{t}\\left[\\frac{\\pi\\_{\\theta}\\left(a\\_{t}\\mid{s\\_{t}}\\right)}{\\pi\\_{\\theta\\_{old}}\\left(a\\_{t}\\mid{s\\_{t}}\\right)})\\hat{A}\\_{t}\\right] = \\hat{\\mathbb{E}}\\_{t}\\left[r\\_{t}\\left(\\theta\\right)\\hat{A}\\_{t}\\right] $$\r\n\r\nWhere $CPI$ refers to a conservative policy iteration. Without a constraint, maximization of $L^{CPI}$ would lead to an excessively large policy update; hence, we PPO modifies the objective, to penalize changes to the policy that move $r\\_{t}\\left(\\theta\\right)$ away from 1:\r\n\r\n$$ J^{\\text{CLIP}}\\left({\\theta}\\right) = \\hat{\\mathbb{E}}\\_{t}\\left[\\min\\left(r\\_{t}\\left(\\theta\\right)\\hat{A}\\_{t}, \\text{clip}\\left(r\\_{t}\\left(\\theta\\right), 1-\\epsilon, 1+\\epsilon\\right)\\hat{A}\\_{t}\\right)\\right] $$\r\n\r\nwhere $\\epsilon$ is a hyperparameter, say, $\\epsilon = 0.2$. The motivation for this objective is as follows. The first term inside the min is $L^{CPI}$. The second term, $\\text{clip}\\left(r\\_{t}\\left(\\theta\\right), 1-\\epsilon, 1+\\epsilon\\right)\\hat{A}\\_{t}$ modifies the surrogate\r\nobjective by clipping the probability ratio, which removes the incentive for moving $r\\_{t}$ outside of the interval $\\left[1 − \\epsilon, 1 + \\epsilon\\right]$. Finally, we take the minimum of the clipped and unclipped objective, so the final objective is a lower bound (i.e., a pessimistic bound) on the unclipped objective. With this scheme, we only ignore the change in probability ratio when it would make the objective improve, and we include it when it makes the objective worse. \r\n\r\nOne detail to note is that when we apply PPO for a network where we have shared parameters for actor and critic functions, we typically add to the objective function an error term on value estimation and an entropy term to encourage exploration.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Proximal Policy Optimization Algorithms","paper":"/paper/proximal-policy-optimization-algorithms","first_author":"John Schulman","n_authors":5,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/proximal-policy-optimization-algorithms"},"source":{"url":"http://arxiv.org/abs/1707.06347v2","title":"Proximal Policy Optimization Algorithms","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Reinforcement Learning","area_id":"reinforcement-learning","collection":"Policy Gradient Methods","url":"/methods/category/policy-gradient-methods","pwc_aliases":[]}],"n_papers_tagged":949,"archive_num_papers":949,"papers_newest_first":[{"paper":null,"title":"AirLLM: Diffusion Policy-based Adaptive LoRA for Remote Fine-Tuning of LLM over the Air","date":"2025-07-15","arxiv_id":"2507.11515","n_code_links":0,"syntology":null},{"paper":"/paper/step-wise-policy-for-rare-tool-knowledge","title":"Step-wise Policy for Rare-tool Knowledge (SPaRK): Offline RL that Drives Diverse Tool Use in LLMs","date":"2025-07-15","arxiv_id":"2507.11371","n_code_links":1,"syntology":null},{"paper":null,"title":"Scene-Aware Conversational ADAS with Generative AI for Real-Time Driver Assistance","date":"2025-07-14","arxiv_id":"2507.10500","n_code_links":0,"syntology":null},{"paper":null,"title":"Meta-Reinforcement Learning for Fast and Data-Efficient Spectrum Allocation in Dynamic Wireless Networks","date":"2025-07-13","arxiv_id":"2507.10619","n_code_links":0,"syntology":null},{"paper":"/paper/deep-reinforcement-learning-with-gradient-1","title":"Deep Reinforcement Learning with Gradient Eligibility Traces","date":"2025-07-12","arxiv_id":"2507.09087","n_code_links":1,"syntology":null},{"paper":"/paper/geo-orbit-a-federated-digital-twin-framework","title":"Geo-ORBIT: A Federated Digital Twin Framework for Scene-Adaptive Lane Geometry Detection","date":"2025-07-11","arxiv_id":"2507.08743","n_code_links":1,"syntology":null},{"paper":null,"title":"LeAD: The LLM Enhanced Planning System Converged with End-to-end Autonomous Driving","date":"2025-07-08","arxiv_id":"2507.05754","n_code_links":0,"syntology":null},{"paper":null,"title":"Model-free Optical Processors using In Situ Reinforcement Learning with Proximal Policy Optimization","date":"2025-07-08","arxiv_id":"2507.05583","n_code_links":0,"syntology":null},{"paper":null,"title":"2048: Reinforcement Learning in a Delayed Reward Environment","date":"2025-07-07","arxiv_id":"2507.05465","n_code_links":0,"syntology":null},{"paper":null,"title":"YOLO-APD: Enhancing YOLOv8 for Robust Pedestrian Detection on Complex Road Geometries","date":"2025-07-07","arxiv_id":"2507.05376","n_code_links":0,"syntology":null},{"paper":null,"title":"LLM-based Realistic Safety-Critical Driving Video Generation","date":"2025-07-02","arxiv_id":"2507.01264","n_code_links":0,"syntology":null},{"paper":null,"title":"BIDA: A Bi-level Interaction Decision-making Algorithm for Autonomous Vehicles in Dynamic Traffic Scenarios","date":"2025-06-19","arxiv_id":"2506.16546","n_code_links":0,"syntology":null},{"paper":null,"title":"Multi-Agent Reinforcement Learning for Autonomous Multi-Satellite Earth Observation: A Realistic Case Study","date":"2025-06-18","arxiv_id":"2506.15207","n_code_links":0,"syntology":null},{"paper":null,"title":"Algorithmic Approaches to Enhance Safety in Autonomous Vehicles: Minimizing Lane Changes and Merging","date":"2025-06-17","arxiv_id":"2506.15026","n_code_links":0,"syntology":null},{"paper":"/paper/light-aircraft-game-basic-implementation-and","title":"Light Aircraft Game : Basic Implementation and training results analysis","date":"2025-06-17","arxiv_id":"2506.14164","n_code_links":1,"syntology":null},{"paper":"/paper/tgdpo-harnessing-token-level-reward-guidance","title":"TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization","date":"2025-06-17","arxiv_id":"2506.14574","n_code_links":1,"syntology":{"ran":2,"of":3,"unverified":1,"pointer_only":3}},{"paper":null,"title":"Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models","date":"2025-06-16","arxiv_id":"2506.13923","n_code_links":0,"syntology":null},{"paper":"/paper/autovla-a-vision-language-action-model-for","title":"AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning","date":"2025-06-16","arxiv_id":"2506.13757","n_code_links":1,"syntology":{"ran":1,"of":1,"unverified":0,"pointer_only":1}},{"paper":null,"title":"How Real is CARLAs Dynamic Vision Sensor? A Study on the Sim-to-Real Gap in Traffic Object Detection","date":"2025-06-16","arxiv_id":"2506.13722","n_code_links":0,"syntology":null},{"paper":null,"title":"Ego-centric Learning of Communicative World Models for Autonomous Driving","date":"2025-06-09","arxiv_id":"2506.08149","n_code_links":0,"syntology":null},{"paper":null,"title":"Autonomous Vehicle Lateral Control Using Deep Reinforcement Learning with MPC-PID Demonstration","date":"2025-06-04","arxiv_id":"2506.04040","n_code_links":0,"syntology":null},{"paper":null,"title":"PPO in the Fisher-Rao geometry","date":"2025-06-04","arxiv_id":"2506.03757","n_code_links":0,"syntology":null},{"paper":"/paper/redrft-a-light-weight-benchmark-for","title":"RedRFT: A Light-Weight Benchmark for Reinforcement Fine-Tuning-Based Red Teaming","date":"2025-06-04","arxiv_id":"2506.04302","n_code_links":1,"syntology":null},{"paper":null,"title":"Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective","date":"2025-06-03","arxiv_id":"2506.02553","n_code_links":0,"syntology":null},{"paper":null,"title":"Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening","date":"2025-06-03","arxiv_id":"2506.02355","n_code_links":0,"syntology":null},{"paper":"/paper/the-surprising-effectiveness-of-negative","title":"The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning","date":"2025-06-02","arxiv_id":"2506.01347","n_code_links":1,"syntology":{"ran":3,"of":3,"unverified":0,"pointer_only":0}},{"paper":null,"title":"DriveMind: A Dual-VLM based Reinforcement Learning Framework for Autonomous Driving","date":"2025-06-01","arxiv_id":"2506.00819","n_code_links":0,"syntology":null},{"paper":null,"title":"Language-Guided Multi-Agent Learning in Simulations: A Unified Framework and Evaluation","date":"2025-06-01","arxiv_id":"2506.04251","n_code_links":0,"syntology":null},{"paper":"/paper/using-diffusion-ensembles-to-estimate","title":"Using Diffusion Ensembles to Estimate Uncertainty for End-to-End Autonomous Driving","date":"2025-05-31","arxiv_id":"2506.00560","n_code_links":0,"syntology":null},{"paper":"/paper/areal-a-large-scale-asynchronous","title":"AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning","date":"2025-05-30","arxiv_id":"2505.24298","n_code_links":1,"syntology":{"ran":0,"of":13,"unverified":13,"pointer_only":0}}],"papers_shown":30,"tasks":[{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":314},{"task":"/task/reinforcement-learning-2","name":"reinforcement-learning","papers":274},{"task":"/task/autonomous-driving","name":"Autonomous Driving","papers":249},{"task":"/task/reinforcement-learning","name":"Reinforcement Learning","papers":242},{"task":"/task/deep-reinforcement-learning","name":"Deep Reinforcement Learning","papers":157},{"task":"/task/autonomous-vehicles","name":"Autonomous Vehicles","papers":83},{"task":"/task/decision-making","name":"Decision Making","papers":65},{"task":"/task/imitation-learning","name":"Imitation Learning","papers":60},{"task":"/task/object-detection","name":"Object Detection","papers":55},{"task":"/task/object-detection-1","name":"object-detection","papers":51},{"task":"/task/continuous-control","name":"Continuous Control","papers":41},{"task":"/task/mujoco","name":"MuJoCo","papers":41},{"task":"/task/continuous-control","name":"continuous-control","papers":37},{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":34},{"task":"/task/language-modelling","name":"Language Modelling","papers":33},{"task":"/task/multi-agent-reinforcement-learning","name":"Multi-agent Reinforcement Learning","papers":28},{"task":"/task/q-learning","name":"Q-Learning","papers":26},{"task":"/task/language-modeling","name":"Language Modeling","papers":24},{"task":"/task/openai-gym","name":"OpenAI Gym","papers":22},{"task":"/task/data-augmentation","name":"Data Augmentation","papers":21}],"tasks_shown":20,"n_tasks":357,"usage_by_year":[{"year":"2017","papers":4},{"year":"2018","papers":21},{"year":"2019","papers":50},{"year":"2020","papers":95},{"year":"2021","papers":113},{"year":"2022","papers":140},{"year":"2023","papers":152},{"year":"2024","papers":232},{"year":"2025","papers":142}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/ppo"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}