{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/understanding-multi-step-deep-reinforcement","title":"Understanding Multi-Step Deep Reinforcement Learning: A Systematic Study of the DQN Target","arxiv_id":"1901.07510","date":"2019-01-22","proceeding":null,"authors":["J. Fernando Hernandez-Garcia","Richard S. Sutton"],"abstract":"Multi-step methods such as Retrace($\\lambda$) and $n$-step $Q$-learning have\nbecome a crucial component of modern deep reinforcement learning agents. These\nmethods are often evaluated as a part of bigger architectures and their\nevaluations rarely include enough samples to draw statistically significant\nconclusions about their performance. This type of methodology makes it\ndifficult to understand how particular algorithmic details of multi-step\nmethods influence learning. In this paper we combine the $n$-step action-value\nalgorithms Retrace, $Q$-learning, Tree Backup, Sarsa, and $Q(\\sigma)$ with an\narchitecture analogous to DQN. We test the performance of all these algorithms\nin the mountain car environment; this choice of environment allows for faster\ntraining times and larger sample sizes. We present statistical analyses on the\neffects of the off-policy correction, the backup length parameter $n$, and the\nupdate frequency of the target network on the performance of these algorithms.\nOur results show that (1) using off-policy correction can have an adverse\neffect on the performance of Sarsa and $Q(\\sigma)$; (2) increasing the backup\nlength $n$ consistently improved performance across all the different\nalgorithms; and (3) the performance of Sarsa and $Q$-learning was more robust\nto the effect of the target network update frequency than the performance of\nTree Backup, $Q(\\sigma)$, and Retrace in this particular task.","url_abs":"http://arxiv.org/abs/1901.07510v2","url_pdf":"http://arxiv.org/pdf/1901.07510v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"understanding-multi-step-deep-reinforcement","repo_url":"https://github.com/kochlisGit/autonomous-vehicles-agent","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"q-learning","task_name":"Q-Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dqn","method_name":"DQN"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"q-learning","method_name":"Q-Learning"},{"method_slug":"retrace","method_name":"Retrace"},{"method_slug":"sarsa","method_name":"Sarsa"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1901.07510","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}