Papers › Recurrent Experience Replay in Distributed Reinforcement Learning

Recurrent Experience Replay in Distributed Reinforcement Learning

1 May 2019ICLR 2019 5archive 2025-07-28

Steven Kapturowski, Georg Ostrovski, Will Dabney, John Quan, Remi Munos

Building on the recent successes of distributed training of RL agents, in this paper we investigate the training of RNN-based RL agents from distributed prioritized experience replay. We study the effects of parameter lag resulting in representational drift and recurrent state staleness and empirically derive an improved training strategy. Using a single network architecture and fixed set of hyperparameters, the resulting agent, Recurrent Replay Distributed DQN, quadruples the previous state of the art on Atari-57, and surpasses the state of the art on DMLab-30. It is the first agent to exceed human-level performance in 52 of the 57 Atari games.

PaperPDFCode

Syntology This paper has no arXiv id in the archive, and this site joins papers to Syntology's graph by arXiv id, so this page does not say whether the graph holds it.

Code

garymm/earl jaxSyntology: not harvested report
michaelnny/deep_rl_zoo pytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Playing Atari with Deep Reinforcement Learning (arXiv:1312.5602; that paper's own run record: 64 ran (of which 24 constructed an object rather than computing a result; 46 with no instrument failure: 5 honoured, 0 violated, 41 with no contract checked; 18 where Syntology's instrument failed) · 53 unverified) and Never Give Up: Learning Directed Exploration Strategies (arXiv:2002.06038; that paper's own run record: 7 ran (of which 3 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 2 where Syntology's instrument failed) · 7 unverified); nothing checks that they implement this paper's method. report
opendilab/DI-engine pytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Hindsight Experience Replay (arXiv:1707.01495; that paper's own run record: 16 ran (of which 10 constructed an object rather than computing a result; 11 with no instrument failure: 1 honoured, 0 violated, 10 with no contract checked; 5 where Syntology's instrument failed) · 0 unverified), Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context (arXiv:1901.02860; that paper's own run record: 67 ran (of which 37 constructed an object rather than computing a result; 49 with no instrument failure: 4 honoured, 1 violated, 44 with no contract checked; 18 where Syntology's instrument failed) · 76 unverified), Decision Transformer: Reinforcement Learning via Sequence Modeling (arXiv:2106.01345; that paper's own run record: 17 ran (of which 10 constructed an object rather than computing a result; 12 with no instrument failure: 0 honoured, 0 violated, 12 with no contract checked; 5 where Syntology's instrument failed) · 9 unverified) and 1 more (every one in this page's JSON twin); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Atari GamesReinforcement LearningReinforcement Learning (RL)reinforcement-learning

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Atari Games Atari 2600 Alien R2D2 Score 229496.9 #4 of 49 Archive leaderboard report
Atari Games Atari 2600 Amidar R2D2 Score 29321.4 #2 of 48 Archive leaderboard report
Atari Games Atari 2600 Assault R2D2 Score 108197.0 #2 of 45 Archive leaderboard report
Atari Games Atari 2600 Asterix R2D2 Score 999153.3 #2 of 49 Archive leaderboard report
Atari Games Atari 2600 Asteroids R2D2 Score 357867.7 #5 of 43 Archive leaderboard report
Atari Games Atari 2600 Atlantis R2D2 Score 1620764.0 #6 of 42 Archive leaderboard report
Atari Games Atari 2600 Bank Heist R2D2 Score 24235.9 #2 of 45 Archive leaderboard report
Atari Games Atari 2600 Battle Zone R2D2 Score 751880.0 #4 of 47 Archive leaderboard report
Atari Games Atari 2600 Beam Rider R2D2 Score 188257.4 #5 of 49 Archive leaderboard report
Atari Games Atari 2600 Berzerk R2D2 Score 53318.7 #5 of 39 Archive leaderboard report
Atari Games Atari 2600 Bowling R2D2 Score 219.5 #4 of 44 Archive leaderboard report
Atari Games Atari 2600 Boxing R2D2 Score 98.5 #20 of 45 Archive leaderboard report
Atari Games Atari 2600 Breakout R2D2 Score 837.7 #7 of 58 Archive leaderboard report
Atari Games Atari 2600 Centipede R2D2 Score 599140.3 #5 of 45 Archive leaderboard report
Atari Games Atari 2600 Chopper Command R2D2 Score 986652.0 #6 of 45 Archive leaderboard report
Atari Games Atari 2600 Crazy Climber R2D2 Score 366690.7 #3 of 49 Archive leaderboard report
Atari Games Atari 2600 Defender R2D2 Score 665792.0 #7 of 21 Archive leaderboard report
Atari Games Atari 2600 Demon Attack R2D2 Score 140002.3 #8 of 46 Archive leaderboard report
Atari Games Atari 2600 Double Dunk R2D2 Score 23.7 #8 of 43 Archive leaderboard report
Atari Games Atari 2600 Enduro R2D2 Score 2372.7 #6 of 48 Archive leaderboard report
Atari Games Atari 2600 Fishing Derby R2D2 Score 85.8 #3 of 44 Archive leaderboard report
Atari Games Atari 2600 Freeway R2D2 Score 32.5 #25 of 59 Archive leaderboard report
Atari Games Atari 2600 Frostbite R2D2 Score 315456.4 #4 of 53 Archive leaderboard report
Atari Games Atari 2600 Gopher R2D2 Score 124776.3 #4 of 43 Archive leaderboard report
Atari Games Atari 2600 Gravitar R2D2 Score 15680.7 #2 of 53 Archive leaderboard report
Atari Games Atari 2600 HERO R2D2 Score 39537.1 #3 of 44 Archive leaderboard report
Atari Games Atari 2600 Ice Hockey R2D2 Score 79.3 #2 of 43 Archive leaderboard report
Atari Games Atari 2600 James Bond R2D2 Score 25354.0 #10 of 45 Archive leaderboard report
Atari Games Atari 2600 Kangaroo R2D2 Score 14130.7 #13 of 47 Archive leaderboard report
Atari Games Atari 2600 Krull R2D2 Score 218448.1 #4 of 45 Archive leaderboard report
Atari Games Atari 2600 Kung-Fu Master R2D2 Score 233413.3 #3 of 43 Archive leaderboard report
Atari Games Atari 2600 Montezuma's Revenge R2D2 Score 2061.3 #18 of 50 Archive leaderboard report
Atari Games Atari 2600 Ms. Pacman R2D2 Score 42281.7 #4 of 47 Archive leaderboard report
Atari Games Atari 2600 Name This Game R2D2 Score 58182.7 #3 of 43 Archive leaderboard report
Atari Games Atari 2600 Phoenix R2D2 Score 864020.0 #6 of 21 Archive leaderboard report
Atari Games Atari 2600 Pitfall! R2D2 Score 0.0 #6 of 23 Archive leaderboard report
Atari Games Atari 2600 Pong R2D2 Score 21.0 #5 of 52 Archive leaderboard report
Atari Games Atari 2600 Private Eye R2D2 Score 5322.7 #13 of 52 Archive leaderboard report
Atari Games Atari 2600 Q*Bert R2D2 Score 408850.0 #3 of 57 Archive leaderboard report
Atari Games Atari 2600 River Raid R2D2 Score 45632.1 #5 of 41 Archive leaderboard report
Atari Games Atari 2600 Road Runner R2D2 Score 599246.7 #5 of 44 Archive leaderboard report
Atari Games Atari 2600 Robotank R2D2 Score 100.4 #7 of 42 Archive leaderboard report
Atari Games Atari 2600 Seaquest R2D2 Score 999996.7 #4 of 57 Archive leaderboard report
Atari Games Atari 2600 Skiing R2D2 Score -30021.7 #5 of 23 Archive leaderboard report
Atari Games Atari 2600 Solaris R2D2 Score 3787.2 #15 of 23 Archive leaderboard report
Atari Games Atari 2600 Space Invaders R2D2 Score 43223.4 #10 of 55 Archive leaderboard report
Atari Games Atari 2600 Star Gunner R2D2 Score 717344.0 #2 of 43 Archive leaderboard report
Atari Games Atari 2600 Surround R2D2 Score 9.9 #3 of 15 Archive leaderboard report
Atari Games Atari 2600 Tennis R2D2 Score -0.1 #39 of 43 Archive leaderboard report
Atari Games Atari 2600 Time Pilot R2D2 Score 445377.3 #3 of 44 Archive leaderboard report
Atari Games Atari 2600 Tutankham R2D2 Score 395.3 #6 of 44 Archive leaderboard report
Atari Games Atari 2600 Up and Down R2D2 Score 589226.9 #8 of 44 Archive leaderboard report
Atari Games Atari 2600 Venture R2D2 Score 1970.7 #8 of 55 Archive leaderboard report
Atari Games Atari 2600 Video Pinball R2D2 Score 999383.2 #1 of 42 Archive leaderboard report
Atari Games Atari 2600 Wizard of Wor R2D2 Score 144362.7 #3 of 41 Archive leaderboard report
Atari Games Atari 2600 Yars Revenge R2D2 Score 995048.4 #2 of 17 Archive leaderboard report
Atari Games Atari 2600 Zaxxon R2D2 Score 224910.7 #3 of 41 Archive leaderboard report
Atari Games Atari games R2D2 Mean Human Normalized Score 3374.31% #6 of 12 Archive leaderboard report
Atari Games Atari-57 R2D2 Human World Record Breakthrough 15 #6 of 11 Archive leaderboard report
Atari Games Atari-57 R2D2 Mean Human Normalized Score 3374.31% #6 of 11 Archive leaderboard report
Atari Games atari game R2D2 Human World Record Breakthrough 15 #6 of 9 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Introduced by this paper: R2D2

ConvolutionDQNDense ConnectionsQ-LearningR2D2

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections