{"url":"/method/td-lambda","slug":"td-lambda","name":"TD Lambda","full_name":"TD Lambda","full_name_withheld":false,"description_markdown":"**TD_INLINE_MATH_1** is a generalisation of **TD_INLINE_MATH_2** reinforcement learning algorithms, but it employs an [eligibility trace](https://paperswithcode.com/method/eligibility-trace) $\\lambda$ and $\\lambda$-weighted returns. The eligibility trace vector is initialized to zero at the beginning of the episode, and it is incremented on each time step by the value gradient, and then fades away by $\\gamma\\lambda$:\r\n\r\n$$ \\textbf{z}\\_{-1} = \\mathbf{0} $$\r\n$$ \\textbf{z}\\_{t} = \\gamma\\lambda\\textbf{z}\\_{t-1} + \\nabla\\hat{v}\\left(S\\_{t}, \\mathbf{w}\\_{t}\\right), 0 \\leq t \\leq T$$\r\n\r\nThe eligibility trace keeps track of which components of the weight vector contribute to recent state valuations. Here $\\nabla\\hat{v}\\left(S\\_{t}, \\mathbf{w}\\_{t}\\right)$ is the feature vector.\r\n\r\nThe TD error for state-value prediction is:\r\n\r\n$$ \\delta\\_{t} = R\\_{t+1} + \\gamma\\hat{v}\\left\\(S\\_{t+1}, \\mathbf{w}\\_{t}\\right) - \\hat{v}\\left(S\\_{t}, \\mathbf{w}\\_{t}\\right) $$\r\n\r\nIn **TD_INLINE_MATH_1**, the weight vector is updated on each step proportional to the scalar TD error and the vector eligibility trace:\r\n\r\n$$ \\mathbf{w}\\_{t+1} = \\mathbf{w}\\_{t} + \\alpha\\delta\\mathbf{z}\\_{t}  $$\r\n\r\nSource: Sutton and Barto, Reinforcement Learning, 2nd Edition","description_state":"present","introduced_year":null,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":null,"title":null,"url_on_a_paper_host":false},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Reinforcement Learning","area_id":"reinforcement-learning","collection":"On-Policy TD Control","url":"/methods/category/on-policy-td-control","pwc_aliases":[]}],"n_papers_tagged":14,"archive_num_papers":14,"papers_newest_first":[{"paper":null,"title":"On-line Policy Improvement using Monte-Carlo Search","date":"2025-01-09","arxiv_id":"2501.05407","n_code_links":0,"syntology":null},{"paper":null,"title":"Model Predictive Control and Reinforcement Learning: A Unified Framework Based on Dynamic Programming","date":"2024-06-02","arxiv_id":"2406.00592","n_code_links":0,"syntology":null},{"paper":"/paper/large-language-models-play-starcraft-ii","title":"Large Language Models Play StarCraft II: Benchmarks and A Chain of Summarization Approach","date":"2023-12-19","arxiv_id":"2312.11865","n_code_links":1,"syntology":{"ran":2,"of":2,"unverified":0,"pointer_only":2}},{"paper":null,"title":"A Robust and Opponent-Aware League Training Method for StarCraft II","date":"2023-09-21","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/alphastar-unplugged-large-scale-offline","title":"AlphaStar Unplugged: Large-Scale Offline Reinforcement Learning","date":"2023-08-07","arxiv_id":"2308.03526","n_code_links":1,"syntology":{"ran":4,"of":4,"unverified":0,"pointer_only":0}},{"paper":"/paper/on-efficient-reinforcement-learning-for-full","title":"On Efficient Reinforcement Learning for Full-length Game of StarCraft II","date":"2022-09-23","arxiv_id":"2209.11553","n_code_links":2,"syntology":{"ran":0,"of":10,"unverified":10,"pointer_only":0}},{"paper":null,"title":"AI in Human-computer Gaming: Techniques, Challenges and Opportunities","date":"2021-11-15","arxiv_id":"2111.07631","n_code_links":0,"syntology":null},{"paper":null,"title":"Search in Imperfect Information Games","date":"2021-11-10","arxiv_id":"2111.05884","n_code_links":0,"syntology":null},{"paper":"/paper/rethinking-of-alphastar","title":"Rethinking of AlphaStar","date":"2021-08-07","arxiv_id":"2108.03452","n_code_links":2,"syntology":null},{"paper":"/paper/an-introduction-of-mini-alphastar","title":"An Introduction of mini-AlphaStar","date":"2021-04-14","arxiv_id":"2104.06890","n_code_links":1,"syntology":null},{"paper":"/paper/deep-reinforcement-learning-with-function","title":"Deep Reinforcement Learning with Function Properties in Mean Reversion Strategies","date":"2021-01-09","arxiv_id":"2101.03418","n_code_links":1,"syntology":null},{"paper":"/paper/tstarbot-x-an-open-sourced-and-comprehensive","title":"TStarBot-X: An Open-Sourced and Comprehensive Study for Efficient League Training in StarCraft II Full Game","date":"2020-11-27","arxiv_id":"2011.13729","n_code_links":1,"syntology":null},{"paper":null,"title":"AlphaStar: An Evolutionary Computation Perspective","date":"2019-02-05","arxiv_id":"1902.01724","n_code_links":0,"syntology":null},{"paper":null,"title":"A Hierarchical Reinforcement Learning Method for Persistent Time-Sensitive Tasks","date":"2016-06-20","arxiv_id":"1606.06355","n_code_links":0,"syntology":null}],"papers_shown":14,"tasks":[{"task":"/task/starcraft-ii","name":"Starcraft II","papers":7},{"task":"/task/starcraft","name":"Starcraft","papers":6},{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":5},{"task":"/task/reinforcement-learning","name":"Reinforcement Learning","papers":4},{"task":"/task/reinforcement-learning-2","name":"reinforcement-learning","papers":4},{"task":"/task/decision-making","name":"Decision Making","papers":2},{"task":"/task/ai-agent","name":"AI Agent","papers":1},{"task":null,"name":"CPU","papers":1},{"task":"/task/deep-reinforcement-learning","name":"Deep Reinforcement Learning","papers":1},{"task":"/task/diversity","name":"Diversity","papers":1},{"task":"/task/hierarchical-reinforcement-learning","name":"Hierarchical Reinforcement Learning","papers":1},{"task":"/task/imitation-learning","name":"Imitation Learning","papers":1},{"task":"/task/language-modelling","name":"Language Modelling","papers":1},{"task":"/task/large-language-model","name":"Large Language Model","papers":1},{"task":"/task/model-predictive-control","name":"Model Predictive Control","papers":1},{"task":"/task/offline-rl","name":"Offline RL","papers":1},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":1}],"tasks_shown":17,"n_tasks":17,"usage_by_year":[{"year":"2016","papers":1},{"year":"2019","papers":1},{"year":"2020","papers":1},{"year":"2021","papers":5},{"year":"2022","papers":1},{"year":"2023","papers":3},{"year":"2024","papers":1},{"year":"2025","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/td-lambda"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}