{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/supervised-policy-update-for-deep","title":"Supervised Policy Update for Deep Reinforcement Learning","arxiv_id":"1805.11706","date":"2018-05-29","proceeding":"ICLR 2019","authors":["Quan Vuong","Yiming Zhang","Keith W. Ross"],"abstract":"We propose a new sample-efficient methodology, called Supervised Policy\nUpdate (SPU), for deep reinforcement learning. Starting with data generated by\nthe current policy, SPU formulates and solves a constrained optimization\nproblem in the non-parameterized proximal policy space. Using supervised\nregression, it then converts the optimal non-parameterized policy to a\nparameterized policy, from which it draws new samples. The methodology is\ngeneral in that it applies to both discrete and continuous action spaces, and\ncan handle a wide variety of proximity constraints for the non-parameterized\noptimization problem. We show how the Natural Policy Gradient and Trust Region\nPolicy Optimization (NPG/TRPO) problems, and the Proximal Policy Optimization\n(PPO) problem can be addressed by this methodology. The SPU implementation is\nmuch simpler than TRPO. In terms of sample efficiency, our extensive\nexperiments show SPU outperforms TRPO in Mujoco simulated robotic tasks and\noutperforms PPO in Atari video game tasks.","url_abs":"http://arxiv.org/abs/1805.11706v4","url_pdf":"http://arxiv.org/pdf/1805.11706v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"supervised-policy-update-for-deep","repo_url":"https://github.com/quanvuong/Supervised_Policy_Update","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"mujoco","task_name":"MuJoCo"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"entropy-regularization","method_name":"Entropy Regularization"},{"method_slug":"ppo","method_name":"PPO"},{"method_slug":"trpo","method_name":"TRPO"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1805.11706","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}