{"url":"/method/muzero","slug":"muzero","name":"MuZero","full_name":"MuZero","full_name_withheld":false,"description_markdown":"**MuZero** is a model-based reinforcement learning algorithm. It builds upon [AlphaZero](https://paperswithcode.com/method/alphazero)'s search and search-based policy iteration algorithms, but incorporates a learned model into the training procedure. \r\n\r\nThe main idea of the algorithm is to predict those aspects of the future that are directly relevant for planning. The model receives the observation (e.g. an image of the Go board or the Atari screen) as an\r\ninput and transforms it into a hidden state. The hidden state is then updated iteratively by a recurrent process that receives the previous hidden state and a hypothetical next action. At every one of these steps the model predicts the policy (e.g. the move to play), value function (e.g. the predicted winner), and immediate reward (e.g. the points scored by playing a move). The model is trained end-to-end, with the sole objective of accurately estimating these three important quantities, so as to match the improved estimates of policy and value generated by search as well as the observed reward. \r\n\r\nThere is no direct constraint or requirement for the hidden state to capture all information necessary to reconstruct the original observation, drastically reducing the amount of information the model has to maintain and predict; nor is there any requirement for the hidden state to match the unknown, true state of the environment; nor any other constraints on the semantics of state. Instead, the hidden states are free to represent state in whatever way is relevant to predicting current and future values and policies. Intuitively, the agent can invent, internally, the rules or dynamics that lead to most accurate planning.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model","paper":"/paper/mastering-atari-go-chess-and-shogi-by","first_author":"Julian Schrittwieser","n_authors":12,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/mastering-atari-go-chess-and-shogi-by"},"source":{"url":"https://arxiv.org/abs/1911.08265v2","title":"Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Reinforcement Learning","area_id":"reinforcement-learning","collection":"Board Game Models","url":"/methods/category/board-game-models","pwc_aliases":[]}],"n_papers_tagged":46,"archive_num_papers":46,"papers_newest_first":[{"paper":null,"title":"Calibrated Value-Aware Model Learning with Stochastic Environment Models","date":"2025-05-28","arxiv_id":"2505.22772","n_code_links":0,"syntology":null},{"paper":"/paper/optionzero-planning-with-learned-options","title":"OptionZero: Planning with Learned Options","date":"2025-02-23","arxiv_id":"2502.16634","n_code_links":1,"syntology":{"ran":7,"of":8,"unverified":1,"pointer_only":8}},{"paper":null,"title":"Reinforcement Learning in Strategy-Based and Atari Games: A Review of Google DeepMinds Innovations","date":"2025-02-14","arxiv_id":"2502.10303","n_code_links":0,"syntology":null},{"paper":null,"title":"Evaluating World Models with LLM for Decision Making","date":"2024-11-13","arxiv_id":"2411.08794","n_code_links":0,"syntology":null},{"paper":null,"title":"Evaluating Robustness of Reinforcement Learning Algorithms for Autonomous Shipping","date":"2024-11-07","arxiv_id":"2411.04915","n_code_links":0,"syntology":null},{"paper":null,"title":"Interpreting the Learned Model in MuZero Planning","date":"2024-11-07","arxiv_id":"2411.04580","n_code_links":0,"syntology":null},{"paper":null,"title":"Combining AI Control Systems and Human Decision Support via Robustness and Criticality","date":"2024-07-03","arxiv_id":"2407.03210","n_code_links":0,"syntology":null},{"paper":"/paper/efficient-monte-carlo-tree-search-via-on-the","title":"Efficient Monte Carlo Tree Search via On-the-Fly State-Conditioned Action Abstraction","date":"2024-06-02","arxiv_id":"2406.00614","n_code_links":1,"syntology":null},{"paper":"/paper/efficient-multi-agent-reinforcement-learning","title":"Efficient Multi-agent Reinforcement Learning by Planning","date":"2024-05-20","arxiv_id":"2405.11778","n_code_links":1,"syntology":null},{"paper":"/paper/rezero-boosting-mcts-based-algorithms-by-just","title":"ReZero: Boosting MCTS-based Algorithms by Backward-view and Entire-buffer Reanalyze","date":"2024-04-25","arxiv_id":"2404.16364","n_code_links":1,"syntology":null},{"paper":"/paper/minizero-comparative-analysis-of-alphazero","title":"MiniZero: Comparative Analysis of AlphaZero and MuZero on Go, Othello, and Atari Games","date":"2023-10-17","arxiv_id":"2310.11305","n_code_links":1,"syntology":{"ran":6,"of":7,"unverified":1,"pointer_only":0}},{"paper":null,"title":"Accelerating Monte Carlo Tree Search with Probability Tree State Abstraction","date":"2023-10-10","arxiv_id":"2310.06513","n_code_links":0,"syntology":null},{"paper":null,"title":"Self-Predictive Universal AI","date":"2023-09-21","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/alphastar-unplugged-large-scale-offline","title":"AlphaStar Unplugged: Large-Scale Offline Reinforcement Learning","date":"2023-08-07","arxiv_id":"2308.03526","n_code_links":1,"syntology":{"ran":4,"of":4,"unverified":0,"pointer_only":0}},{"paper":null,"title":"$λ$-models: Effective Decision-Aware Reinforcement Learning with Latent Models","date":"2023-06-30","arxiv_id":"2306.17366","n_code_links":0,"syntology":null},{"paper":null,"title":"What model does MuZero learn?","date":"2023-06-01","arxiv_id":"2306.00840","n_code_links":0,"syntology":null},{"paper":"/paper/model-predictive-control-with-self-supervised","title":"Model Predictive Control with Self-supervised Representation Learning","date":"2023-04-14","arxiv_id":"2304.07219","n_code_links":1,"syntology":null},{"paper":null,"title":"Equivariant MuZero","date":"2023-02-09","arxiv_id":"2302.04798","n_code_links":0,"syntology":null},{"paper":null,"title":"Epistemic Monte Carlo Tree Search","date":"2022-10-21","arxiv_id":"2210.13455","n_code_links":0,"syntology":null},{"paper":"/paper/efficient-offline-policy-optimization-with-a","title":"Efficient Offline Policy Optimization with a Learned Model","date":"2022-10-12","arxiv_id":"2210.05980","n_code_links":1,"syntology":null},{"paper":"/paper/human-level-atari-200x-faster","title":"Human-level Atari 200x faster","date":"2022-09-15","arxiv_id":"2209.07550","n_code_links":1,"syntology":{"ran":1,"of":3,"unverified":2,"pointer_only":0}},{"paper":"/paper/transformers-are-sample-efficient-world","title":"Transformers are Sample-Efficient World Models","date":"2022-09-01","arxiv_id":"2209.00588","n_code_links":2,"syntology":{"ran":17,"of":26,"unverified":9,"pointer_only":26}},{"paper":null,"title":"An AlphaZero-Inspired Approach to Solving Search Problems","date":"2022-07-02","arxiv_id":"2207.00919","n_code_links":0,"syntology":null},{"paper":"/paper/impartial-games-a-challenge-for-reinforcement","title":"Impartial Games: A Challenge for Reinforcement Learning","date":"2022-05-25","arxiv_id":"2205.12787","n_code_links":1,"syntology":null},{"paper":"/paper/towards-evaluating-adaptivity-of-model-based","title":"Towards Evaluating Adaptivity of Model-Based Reinforcement Learning Methods","date":"2022-04-25","arxiv_id":"2204.11464","n_code_links":1,"syntology":null},{"paper":"/paper/reward-reports-for-reinforcement-learning","title":"Reward Reports for Reinforcement Learning","date":"2022-04-22","arxiv_id":"2204.10817","n_code_links":1,"syntology":null},{"paper":null,"title":"MuZero with Self-competition for Rate Control in VP9 Video Compression","date":"2022-02-14","arxiv_id":"2202.06626","n_code_links":0,"syntology":null},{"paper":"/paper/procedural-generalization-by-planning-with-1","title":"Procedural Generalization by Planning with Self-Supervised World Models","date":"2021-11-02","arxiv_id":"2111.01587","n_code_links":0,"syntology":null},{"paper":"/paper/mastering-atari-games-with-limited-data","title":"Mastering Atari Games with Limited Data","date":"2021-10-30","arxiv_id":"2111.00210","n_code_links":3,"syntology":{"ran":2,"of":3,"unverified":1,"pointer_only":0}},{"paper":"/paper/planning-in-stochastic-environments-with-a","title":"Planning in Stochastic Environments with a Learned Model","date":"2021-09-29","arxiv_id":null,"n_code_links":2,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/reinforcement-learning-2","name":"reinforcement-learning","papers":21},{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":19},{"task":"/task/model-based-reinforcement-learning","name":"Model-based Reinforcement Learning","papers":18},{"task":"/task/reinforcement-learning","name":"Reinforcement Learning","papers":15},{"task":"/task/atari-games","name":"Atari Games","papers":8},{"task":"/task/board-games","name":"Board Games","papers":7},{"task":"/task/decision-making","name":"Decision Making","papers":6},{"task":"/task/continuous-control","name":"Continuous Control","papers":5},{"task":"/task/deep-reinforcement-learning","name":"Deep Reinforcement Learning","papers":5},{"task":"/task/game-of-go","name":"Game of Go","papers":5},{"task":"/task/continuous-control","name":"continuous-control","papers":4},{"task":"/task/model","name":"model","papers":4},{"task":"/task/atari-games-100k","name":"Atari Games 100k","papers":3},{"task":"/task/offline-rl","name":"Offline RL","papers":3},{"task":"/task/representation-learning","name":"Representation Learning","papers":3},{"task":"/task/computational-efficiency","name":"Computational Efficiency","papers":2},{"task":"/task/meta-learning","name":"Meta-Learning","papers":2},{"task":"/task/starcraft","name":"Starcraft","papers":2},{"task":"/task/benchmarking","name":"Benchmarking","papers":1},{"task":"/task/chatbot","name":"Chatbot","papers":1}],"tasks_shown":20,"n_tasks":38,"usage_by_year":[{"year":"2019","papers":1},{"year":"2020","papers":5},{"year":"2021","papers":13},{"year":"2022","papers":9},{"year":"2023","papers":8},{"year":"2024","papers":7},{"year":"2025","papers":3}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/muzero"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}