{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/true-online-temporal-difference-learning","title":"True Online Temporal-Difference Learning","arxiv_id":"1512.04087","date":"2015-12-13","proceeding":null,"authors":["Harm van Seijen","A. Rupam Mahmood","Patrick M. Pilarski","Marlos C. Machado","Richard S. Sutton"],"abstract":"The temporal-difference methods TD($\\lambda$) and Sarsa($\\lambda$) form a\ncore part of modern reinforcement learning. Their appeal comes from their good\nperformance, low computational cost, and their simple interpretation, given by\ntheir forward view. Recently, new versions of these methods were introduced,\ncalled true online TD($\\lambda$) and true online Sarsa($\\lambda$), respectively\n(van Seijen & Sutton, 2014). These new versions maintain an exact equivalence\nwith the forward view at all times, whereas the traditional versions only\napproximate it for small step-sizes. We hypothesize that these true online\nmethods not only have better theoretical properties, but also dominate the\nregular methods empirically. In this article, we put this hypothesis to the\ntest by performing an extensive empirical comparison. Specifically, we compare\nthe performance of true online TD($\\lambda$)/Sarsa($\\lambda$) with regular\nTD($\\lambda$)/Sarsa($\\lambda$) on random MRPs, a real-world myoelectric\nprosthetic arm, and a domain from the Arcade Learning Environment. We use\nlinear function approximation with tabular, binary, and non-binary features.\nOur results suggest that the true online methods indeed dominate the regular\nmethods. Across all domains/representations the learning speed of the true\nonline methods are often better, but never worse than that of the regular\nmethods. An additional advantage is that no choice between traces has to be\nmade for the true online methods. Besides the empirical results, we provide an\nin-depth analysis of the theory behind true online temporal-difference\nlearning. In addition, we show that new true online temporal-difference methods\ncan be derived by making changes to the online forward view and then rewriting\nthe update equations.","url_abs":"http://arxiv.org/abs/1512.04087v2","url_pdf":"http://arxiv.org/pdf/1512.04087v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"true-online-temporal-difference-learning","repo_url":"https://github.com/armahmood/totd-rndmdp-experiments","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"atari-games","task_name":"Atari Games"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"}],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1512.04087","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}