{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/asynchronous-episodic-deep-deterministic","title":"Asynchronous Episodic Deep Deterministic Policy Gradient: Towards Continuous Control in Computationally Complex Environments","arxiv_id":"1903.00827","date":"2019-03-03","proceeding":null,"authors":["Zhizheng Zhang","Jiale Chen","Zhibo Chen","Weiping Li"],"abstract":"Deep Deterministic Policy Gradient (DDPG) has been proved to be a successful\nreinforcement learning (RL) algorithm for continuous control tasks. However,\nDDPG still suffers from data insufficiency and training inefficiency,\nespecially in computationally complex environments. In this paper, we propose\nAsynchronous Episodic DDPG (AE-DDPG), as an expansion of DDPG, which can\nachieve more effective learning with less training time required. First, we\ndesign a modified scheme for data collection in an asynchronous fashion.\nGenerally, for asynchronous RL algorithms, sample efficiency or/and training\nstability diminish as the degree of parallelism increases. We consider this\nproblem from the perspectives of both data generation and data utilization. In\ndetail, we re-design experience replay by introducing the idea of episodic\ncontrol so that the agent can latch on good trajectories rapidly. In addition,\nwe also inject a new type of noise in action space to enrich the exploration\nbehaviors. Experiments demonstrate that our AE-DDPG achieves higher rewards and\nrequires less time consuming than most popular RL algorithms in Learning to Run\ntask which has a computationally complex environment. Not limited to the\ncontrol tasks in computationally complex environments, AE-DDPG also achieves\nhigher rewards and 2- to 4-fold improvement in sample efficiency on average\ncompared to other variants of DDPG in MuJoCo environments. Furthermore, we\nverify the effectiveness of each proposed technique component through abundant\nablation study.","url_abs":"http://arxiv.org/abs/1903.00827v1","url_pdf":"http://arxiv.org/pdf/1903.00827v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"asynchronous-episodic-deep-deterministic","repo_url":"https://github.com/anita-hu/TF2-RL","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"mujoco","task_name":"MuJoCo"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"continuous-control","task_name":"continuous-control"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"ddpg","method_name":"DDPG"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"experience-replay","method_name":"Experience Replay"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1903.00827","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}