{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/autoregressive-policies-for-continuous","title":"Autoregressive Policies for Continuous Control Deep Reinforcement Learning","arxiv_id":"1903.11524","date":"2019-03-27","proceeding":null,"authors":["Dmytro Korenkevych","A. Rupam Mahmood","Gautham Vasan","James Bergstra"],"abstract":"Reinforcement learning algorithms rely on exploration to discover new\nbehaviors, which is typically achieved by following a stochastic policy. In\ncontinuous control tasks, policies with a Gaussian distribution have been\nwidely adopted. Gaussian exploration however does not result in smooth\ntrajectories that generally correspond to safe and rewarding behaviors in\npractical tasks. In addition, Gaussian policies do not result in an effective\nexploration of an environment and become increasingly inefficient as the action\nrate increases. This contributes to a low sample efficiency often observed in\nlearning continuous control tasks. We introduce a family of stationary\nautoregressive (AR) stochastic processes to facilitate exploration in\ncontinuous control domains. We show that proposed processes possess two\ndesirable features: subsequent process observations are temporally coherent\nwith continuously adjustable degree of coherence, and the process stationary\ndistribution is standard normal. We derive an autoregressive policy (ARP) that\nimplements such processes maintaining the standard agent-environment interface.\nWe show how ARPs can be easily used with the existing off-the-shelf learning\nalgorithms. Empirically we demonstrate that using ARPs results in improved\nexploration and sample efficiency in both simulated and real world domains,\nand, furthermore, provides smooth exploration trajectories that enable safe\noperation of robotic hardware.","url_abs":"http://arxiv.org/abs/1903.11524v1","url_pdf":"http://arxiv.org/pdf/1903.11524v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"autoregressive-policies-for-continuous","repo_url":"https://github.com/kindredresearch/arp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"continuous-control","task_name":"continuous-control"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1903.11524","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}