{"url":"/method/prioritized-experience-replay","slug":"prioritized-experience-replay","name":"Prioritized Experience Replay","full_name":"Prioritized Experience Replay","full_name_withheld":false,"description_markdown":"**Prioritized Experience Replay** is a type of [experience replay](https://paperswithcode.com/method/experience-replay) in reinforcement learning where we more frequently replay transitions with high expected learning progress, as measured by the magnitude of their temporal-difference (TD) error. This prioritization can lead to a loss of diversity, which is alleviated with stochastic prioritization, and introduce bias, which can be corrected with importance sampling.\r\n\r\nThe stochastic sampling method interpolates between pure greedy prioritization and uniform random sampling. The probability of being sampled is ensured to be monotonic in a transition's priority,  while guaranteeing a non-zero probability even for the lowest-priority transition. Concretely, define the probability of sampling transition $i$ as\r\n\r\n$$P(i) = \\frac{p_i^{\\alpha}}{\\sum_k p_k^{\\alpha}}$$\r\n\r\nwhere $p_i > 0$ is the priority of transition $i$. The exponent $\\alpha$ determines how much prioritization is used, with $\\alpha=0$ corresponding to the uniform case.\r\n\r\nPrioritized replay introduces bias because it changes this distribution in an uncontrolled fashion, and therefore changes the solution that the estimates will converge to. We can correct this bias by using\r\nimportance-sampling (IS) weights:\r\n\r\n$$ w\\_{i} = \\left(\\frac{1}{N}\\cdot\\frac{1}{P\\left(i\\right)}\\right)^{\\beta} $$\r\n\r\nthat fully compensates for the non-uniform probabilities $P\\left(i\\right)$ if $\\beta = 1$. These weights can be folded into the [Q-learning](https://paperswithcode.com/method/q-learning) update by using $w\\_{i}\\delta\\_{i}$ instead of $\\delta\\_{i}$ - weighted IS rather than ordinary IS. For stability reasons, we always normalize weights by $1/\\max\\_{i}w\\_{i}$ so\r\nthat they only scale the update downwards.\r\n\r\nThe two types of prioritization are proportional based, where $p\\_{i} = |\\delta\\_{i}| + \\epsilon$ and rank-based, where $p\\_{i} = \\frac{1}{\\text{rank}\\left(i\\right)}$, the latter where $\\text{rank}\\left(i\\right)$ is the rank of transition $i$ when the replay memory is sorted according to |$\\delta\\_{i}$|, For proportional based, hyperparameters used were $\\alpha = 0.7$, $\\beta\\_{0} = 0.5$. For the rank-based variant, hyperparameters used were $\\alpha = 0.6$, $\\beta\\_{0} = 0.4$.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Prioritized Experience Replay","paper":"/paper/prioritized-experience-replay","first_author":"Tom Schaul","n_authors":4,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/prioritized-experience-replay"},"source":{"url":"http://arxiv.org/abs/1511.05952v4","title":"Prioritized Experience Replay","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Reinforcement Learning","area_id":"reinforcement-learning","collection":"Replay Memory","url":"/methods/category/replay-memory","pwc_aliases":[]}],"n_papers_tagged":138,"archive_num_papers":138,"papers_newest_first":[{"paper":"/paper/cawr-corruption-averse-advantage-weighted","title":"CAWR: Corruption-Averse Advantage-Weighted Regression for Robust Policy Optimization","date":"2025-06-18","arxiv_id":"2506.15654","n_code_links":1,"syntology":null},{"paper":null,"title":"Calibrated Value-Aware Model Learning with Stochastic Environment Models","date":"2025-05-28","arxiv_id":"2505.22772","n_code_links":0,"syntology":null},{"paper":"/paper/online-learning-based-adaptive-beam-switching","title":"Online Learning-based Adaptive Beam Switching for 6G Networks: Enhancing Efficiency and Resilience","date":"2025-05-12","arxiv_id":"2505.08032","n_code_links":1,"syntology":null},{"paper":null,"title":"Graph Based Deep Reinforcement Learning Aided by Transformers for Multi-Agent Cooperation","date":"2025-04-11","arxiv_id":"2504.08195","n_code_links":0,"syntology":null},{"paper":null,"title":"PER-DPP Sampling Framework and Its Application in Path Planning","date":"2025-03-10","arxiv_id":"2503.07411","n_code_links":0,"syntology":null},{"paper":"/paper/optionzero-planning-with-learned-options","title":"OptionZero: Planning with Learned Options","date":"2025-02-23","arxiv_id":"2502.16634","n_code_links":1,"syntology":{"ran":7,"of":8,"unverified":1,"pointer_only":8}},{"paper":null,"title":"Reinforcement Learning in Strategy-Based and Atari Games: A Review of Google DeepMinds Innovations","date":"2025-02-14","arxiv_id":"2502.10303","n_code_links":0,"syntology":null},{"paper":null,"title":"Enhancing UAV Path Planning Efficiency Through Accelerated Learning","date":"2025-01-17","arxiv_id":"2501.10141","n_code_links":0,"syntology":null},{"paper":null,"title":"SALE-Based Offline Reinforcement Learning with Ensemble Q-Networks","date":"2025-01-07","arxiv_id":"2501.03676","n_code_links":0,"syntology":null},{"paper":null,"title":"Evaluating World Models with LLM for Decision Making","date":"2024-11-13","arxiv_id":"2411.08794","n_code_links":0,"syntology":null},{"paper":null,"title":"Evaluating Robustness of Reinforcement Learning Algorithms for Autonomous Shipping","date":"2024-11-07","arxiv_id":"2411.04915","n_code_links":0,"syntology":null},{"paper":null,"title":"Interpreting the Learned Model in MuZero Planning","date":"2024-11-07","arxiv_id":"2411.04580","n_code_links":0,"syntology":null},{"paper":"/paper/beyond-the-rainbow-high-performance-deep","title":"Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC","date":"2024-11-06","arxiv_id":"2411.03820","n_code_links":3,"syntology":{"ran":13,"of":25,"unverified":12,"pointer_only":21}},{"paper":null,"title":"Enhancing LLM Agents for Code Generation with Possibility and Pass-rate Prioritized Experience Replay","date":"2024-10-16","arxiv_id":"2410.12236","n_code_links":0,"syntology":null},{"paper":null,"title":"Learning in complex action spaces without policy gradients","date":"2024-10-08","arxiv_id":"2410.06317","n_code_links":0,"syntology":null},{"paper":null,"title":"Investigating the Interplay of Prioritized Replay and Generalization","date":"2024-07-12","arxiv_id":"2407.09702","n_code_links":0,"syntology":null},{"paper":"/paper/roer-regularized-optimal-experience-replay","title":"ROER: Regularized Optimal Experience Replay","date":"2024-07-04","arxiv_id":"2407.03995","n_code_links":1,"syntology":null},{"paper":null,"title":"Combining AI Control Systems and Human Decision Support via Robustness and Criticality","date":"2024-07-03","arxiv_id":"2407.03210","n_code_links":0,"syntology":null},{"paper":null,"title":"Physics-informed Imitative Reinforcement Learning for Real-world Driving","date":"2024-06-18","arxiv_id":"2407.02508","n_code_links":0,"syntology":null},{"paper":"/paper/efficient-monte-carlo-tree-search-via-on-the","title":"Efficient Monte Carlo Tree Search via On-the-Fly State-Conditioned Action Abstraction","date":"2024-06-02","arxiv_id":"2406.00614","n_code_links":1,"syntology":null},{"paper":null,"title":"Shared-unique Features and Task-aware Prioritized Sampling on Multi-task Reinforcement Learning","date":"2024-06-02","arxiv_id":"2406.00761","n_code_links":0,"syntology":null},{"paper":"/paper/efficient-multi-agent-reinforcement-learning","title":"Efficient Multi-agent Reinforcement Learning by Planning","date":"2024-05-20","arxiv_id":"2405.11778","n_code_links":1,"syntology":null},{"paper":"/paper/rezero-boosting-mcts-based-algorithms-by-just","title":"ReZero: Boosting MCTS-based Algorithms by Backward-view and Entire-buffer Reanalyze","date":"2024-04-25","arxiv_id":"2404.16364","n_code_links":1,"syntology":null},{"paper":"/paper/laser-learning-environment-a-new-environment","title":"Laser Learning Environment: A new environment for coordination-critical multi-agent tasks","date":"2024-04-04","arxiv_id":"2404.03596","n_code_links":1,"syntology":null},{"paper":null,"title":"An Improved Strategy for Blood Glucose Control Using Multi-Step Deep Reinforcement Learning","date":"2024-03-12","arxiv_id":"2403.07566","n_code_links":0,"syntology":null},{"paper":null,"title":"Reinforcement learning to maximise wind turbine energy generation","date":"2024-02-17","arxiv_id":"2402.11384","n_code_links":0,"syntology":null},{"paper":null,"title":"Directly Attention Loss Adjusted Prioritized Experience Replay","date":"2023-11-24","arxiv_id":"2311.14390","n_code_links":0,"syntology":null},{"paper":null,"title":"Multi-Timescale Control and Communications with Deep Reinforcement Learning -- Part II: Control-Aware Radio Resource Allocation","date":"2023-11-19","arxiv_id":"2311.11280","n_code_links":0,"syntology":null},{"paper":null,"title":"Advancing Algorithmic Trading: A Multi-Technique Enhancement of Deep Q-Network Models","date":"2023-11-09","arxiv_id":"2311.05743","n_code_links":0,"syntology":null},{"paper":null,"title":"Mitigating Estimation Errors by Twin TD-Regularized Actor and Critic for Deep Reinforcement Learning","date":"2023-11-07","arxiv_id":"2311.03711","n_code_links":0,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":72},{"task":"/task/reinforcement-learning-2","name":"reinforcement-learning","papers":68},{"task":"/task/reinforcement-learning","name":"Reinforcement Learning","papers":60},{"task":"/task/deep-reinforcement-learning","name":"Deep Reinforcement Learning","papers":41},{"task":"/task/model-based-reinforcement-learning","name":"Model-based Reinforcement Learning","papers":20},{"task":"/task/atari-games","name":"Atari Games","papers":17},{"task":"/task/q-learning","name":"Q-Learning","papers":17},{"task":"/task/continuous-control","name":"Continuous Control","papers":15},{"task":"/task/continuous-control","name":"continuous-control","papers":14},{"task":"/task/decision-making","name":"Decision Making","papers":11},{"task":"/task/board-games","name":"Board Games","papers":8},{"task":"/task/openai-gym","name":"OpenAI Gym","papers":8},{"task":"/task/multi-agent-reinforcement-learning","name":"Multi-agent Reinforcement Learning","papers":7},{"task":"/task/game-of-go","name":"Game of Go","papers":6},{"task":"/task/mujoco","name":"MuJoCo","papers":5},{"task":"/task/offline-rl","name":"Offline RL","papers":5},{"task":"/task/starcraft","name":"Starcraft","papers":5},{"task":"/task/model","name":"model","papers":5},{"task":"/task/autonomous-driving","name":"Autonomous Driving","papers":4},{"task":"/task/atari-games-100k","name":"Atari Games 100k","papers":3}],"tasks_shown":20,"n_tasks":82,"usage_by_year":[{"year":"2015","papers":1},{"year":"2017","papers":2},{"year":"2018","papers":8},{"year":"2019","papers":10},{"year":"2020","papers":22},{"year":"2021","papers":26},{"year":"2022","papers":21},{"year":"2023","papers":22},{"year":"2024","papers":17},{"year":"2025","papers":9}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/prioritized-experience-replay"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}