Papers › Distributed Prioritized Experience Replay

Distributed Prioritized Experience Replay

2 Mar 2018ICLR 2018 1arXiv:1803.00933archive 2025-07-28

Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado van Hasselt, David Silver

We propose a distributed architecture for deep reinforcement learning at scale, that enables agents to learn effectively from orders of magnitude more data than previously possible. The algorithm decouples acting from learning: the actors interact with their own instances of the environment by selecting actions according to a shared neural network, and accumulate the resulting experience in a shared experience replay memory; the learner replays samples of experience and updates the neural network. The architecture relies on prioritized experience replay to focus only on the most significant data generated by the actors. Our architecture substantially improves the state of the art on the Arcade Learning Environment, achieving better final performance in a fraction of the wall-clock training time.

PaperPDFConference PDFCodeCode Syntology ran

In Syntology View this paper on Syntology, its page in Syntology's graph. That page adds which of its functions are also present in other papers' code Syntology holds, with dates, and its dated run record as JSON and a README badge. It also lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

Syntology 15 code samples harvested from 3 repositories linked to this paper: 9 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 0 honoured, 0 violated, 9 with no contract checked; 0 where Syntology's instrument failed) · 6 unverified. “Where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code; here there are none.

By repository: community (archive-listed): 15 samples from 3 repositories, 9 ran (9 with no instrument failure; 0 where Syntology's instrument failed). The run record, sample by sample. “Ran” means executed on a synthesized input, not that the code is correct or reproduces the paper.

No sample is linked to this paper from 4 of its repositories that Syntology harvested for other papers:

Levels written on 2026-09-26 (UTC): the dates Syntology's graph wrote each sample's verification level, not when the code executed.

15 repositories listed; official and paper-mentioned ones first.

HussonnoisMaxence/RL_Algorithms mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Deep Reinforcement Learning with Double Q-learning (arXiv:1509.06461; that paper's own run record: 56 ran (of which 38 constructed an object rather than computing a result; 55 with no instrument failure: 0 honoured, 0 violated, 55 with no contract checked; 1 where Syntology's instrument failed) · 50 unverified) and Prioritized Experience Replay (arXiv:1511.05952; that paper's own run record: 80 ran (of which 62 constructed an object rather than computing a result; 72 with no instrument failure: 4 honoured, 0 violated, 68 with no contract checked; 8 where Syntology's instrument failed) · 31 unverified); nothing checks that they implement this paper's method. report
Lyusungwon/apex_dqn_pytorch mentioned on GitHubpytorchSyntology: not harvested report
belepi93/Ape-X mentioned on GitHubpytorchSyntology: not harvested report
cindycia/Atari-SAC-Discrete mentioned on GitHubpytorchMITSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Soft Actor-Critic for Discrete Action Settings (arXiv:1910.07207; that paper's own run record: 12 ran (of which 0 constructed an object rather than computing a result; 12 with no instrument failure: 2 honoured, 1 violated, 9 with no contract checked; 0 where Syntology's instrument failed) · 6 unverified); nothing checks that they implement this paper's method. report
dannysdeng/dqn-pytorch mentioned on GitHubpytorchSyntology: not harvested report
eladsar/rbi mentioned on GitHubpytorchSyntology: not harvested report
haje01/distper mentioned on GitHubpytorchMITSyntology: 3 samples, 3 ran (0 instrument-failed) · 0 unverified report
ku2482/rltorch mentioned on GitHubpytorchMITSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Soft Actor-Critic for Discrete Action Settings (arXiv:1910.07207; that paper's own run record: 12 ran (of which 0 constructed an object rather than computing a result; 12 with no instrument failure: 2 honoured, 1 violated, 9 with no contract checked; 0 where Syntology's instrument failed) · 6 unverified); nothing checks that they implement this paper's method. report
mightypirate1/DRL-Tetris mentioned on GitHubtfGPL-3.0Syntology: not harvested report
neka-nat/distributed_rl mentioned on GitHubpytorchMITSyntology: 4 samples, 4 ran (0 instrument-failed) · 0 unverified report
sherry4186/Distributed-DQN mentioned on GitHubSyntology: not harvested report
uber-research/ape-x mentioned on GitHubtfApache-2.0Syntology: 8 samples, 2 ran (0 instrument-failed) · 6 unverified report
vwxyzjn/cleanrl mentioned on GitHubpytorchNOASSERTIONSyntology: not harvested report
opendilab/DI-engine pytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Hindsight Experience Replay (arXiv:1707.01495; that paper's own run record: 16 ran (of which 10 constructed an object rather than computing a result; 11 with no instrument failure: 1 honoured, 0 violated, 10 with no contract checked; 5 where Syntology's instrument failed) · 0 unverified), Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context (arXiv:1901.02860; that paper's own run record: 67 ran (of which 37 constructed an object rather than computing a result; 49 with no instrument failure: 4 honoured, 1 violated, 44 with no contract checked; 18 where Syntology's instrument failed) · 76 unverified), Decision Transformer: Reinforcement Learning via Sequence Modeling (arXiv:2106.01345; that paper's own run record: 17 ran (of which 10 constructed an object rather than computing a result; 12 with no instrument failure: 0 honoured, 0 violated, 12 with no contract checked; 5 where Syntology's instrument failed) · 9 unverified) and 1 more (every one in this page's JSON twin); nothing checks that they implement this paper's method. report
ray-project/ray Syntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

15 samples harvested: 9 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 0 honoured, 0 violated, 9 with no contract checked; 0 where Syntology's instrument failed) · 6 unverified. “Where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Syntology's record for this page has not changed since , the first build that kept a record date for it; when this build read Syntology's graph is in the build record. That date is not when the samples ran.

9ran
6unverified

Licence: 3 of the 15 samples are pointer only, meaning Syntology does not serve that copy's text. For 3 of them, this paper's copy is under a permissive licence and cleared, but the stored copy of the same code that Syntology serves is not; the licence label's tooltip gives Syntology's recorded reason. This page shows no code text for any sample; each one links to its file in the repository.

Harvested from 3 repositories linked to this paper, official or community; the list below is grouped by repository, each group headed by its name and what it is to the paper. “Ran” means the sample executed on a synthesized input. It does not mean the output is correct, and nothing here reproduces the paper's results. A line beginning “fixture:” under a sample that ran gives the arguments Syntology's fixture called the sample with, each as name:type/rank/element kind (? where none was recorded), and the type of what the fixture's run returned, exactly as the harness recorded them; the arguments and the returned type were recorded by two separate runs of the same fixture. No output value is shown, and a returned type says nothing about whether the output was right. “Level recorded” gives the UTC date when Syntology's graph wrote the sample's verification level, not when the code executed; where the level was written before Syntology's graph kept that date, the row says “level recorded, date not recorded”. “Honoured” and “violated” refer to a contract Syntology drafted from the code itself. “Our draft was wrong” and “the contract check could not run” are failures of Syntology's instrument, not of the code: the sample ran on Syntology's synthesized input, and then the drafted contract did not describe it, or the check itself raised an error (RAISES), lacked a dependency (DEP_MISSING) or ran out of time (TIMEOUT); hover the label for which. A metamorphic tier, where shown, is evidence of the output's shape, not that the sample implements the method.

Each sample ends with its code_sha256. A sample's code_sha256 is Syntology's identity for that exact code. It is printed in full (16 hexadecimal characters for every sample in this build), and get_code(code_sha256="…") takes it exactly as printed. An agent calls it on Syntology's MCP service (how to connect); click an identity to copy that call.

Repository labels, per sample. official repository: The archive marks this repository official for the paper. named in the paper: The archive records that the paper mentions this repository; it is not marked official. community (archive-listed): In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper. found in paper text by Syntology: Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted. community: Not in the archive's code links for this paper; a community repository Syntology harvested. Samples are grouped by repository: a repository marked official first, then by how directly the repository is tied to the paper, then in the archive's order; never by how many ran. Licence labels name the repository's licence as recorded at harvest. “Pointer only” means Syntology does not serve that copy's text, for one of six reasons: no licence file was found; the licence was not identified; the licence is recorded as permissive but that copy's record is not marked cleared; the licence is outside the permissive list Syntology serves text under (MIT, Apache-2.0, BSD and similar); this paper's copy was not located, so its licence is not recorded; or this paper's copy is cleared, but the stored copy of the same code that Syntology serves is not. Some licences outside that list permit redistribution, such as WTFPL, and GPL-3.0 under its conditions; they are simply not on the list. Hover a licence label for the reason. File links open the file on GitHub at the default branch, which may have changed since the harvest.

A function or class name links to its page when the name occurs in the harvested code of at least 5 papers; the name is a string, not an identity. A census tag appears when Syntology's census ran implementations with that name on one shared input per bucket: it gives how many were compared, from how many papers, and how many distinct outputs they produced, and names this sample's bucket when this sample was one of them.

haje01/distper community (archive-listed) 3 samples, 3 ran (0 instrument-failed) · 0 unverified
float2byte common.py ran fingerprinted MIT (permissive) · 2731616644b0f64e · report fixture: signature data:ndarray/2/float · returned ndarray · level recorded 2026-09-26
get_size in 26 papers common.py ran MIT (permissive) · 8d1439744144727d · report fixture: signature obj:dict/0/?, seen:NoneType/0/? · returned int · level recorded 2026-09-26
make_env in 50 papers wrappers.py ran MIT (permissive) · 94cca620996f3fbd · report fixture: signature env_name:str/0/str · returned ScaledFloatFrame · level recorded 2026-09-26
neka-nat/distributed_rl community (archive-listed) 4 samples, 4 ran (0 instrument-failed) · 0 unverified
epsilon_greedy distributed_rl/libs/utils.py ran MIT (permissive) · f72bfc9bc83789b4 · report fixture: signature state:Tensor/2/float, policy_net:callable/0/?, eps:float/0/float · returned Tensor · level recorded 2026-09-26
generate_deque distributed_rl/libs/replay_memory.py ran MIT (permissive) · 2f9dde3cc529335b · report fixture: signature use_compress:bool/0/bool, capacity:int/0/int · returned CompressedDeque · level recorded 2026-09-26
outsize distributed_rl/libs/utils.py ran MIT (permissive) · 55903738c3c01f23 · report fixture: signature x:tuple/1/int, f:int/0/int, p:int/0/int, s:int/0/int · returned tuple · level recorded 2026-09-26
preprocess in 284 papers
census: 11 implementations from 13 papers, 10 distinct outputs across 5 buckets, one shared input per bucket
distributed_rl/libs/utils.py ran MIT (permissive) · bf087e46257f7648 · report fixture: signature img:ndarray/3/int, shape:tuple/1/int, gray:bool/0/bool · returned ndarray · level recorded 2026-09-26
uber-research/ape-x community (archive-listed) 8 samples, 2 ran (0 instrument-failed) · 6 unverified
get_game_obs gym_tensorflow/atari/tf_atari.py ran Apache-2.0 (permissive) · 62e7940355c01ad9 · report fixture: signature game:str/0/str · returned tuple · level recorded 2026-09-26
make_masked_frame replay_buffer_actor.py ran Apache-2.0 (permissive) · 3fd5d2d717b7f8bc · report fixture: signature frames:list/1/EagerTensor, dones:list/1/EagerTensor, data_format:str/0/str · returned EagerTensor · level recorded 2026-09-26
build_act in 5 papers apex.py unverified Apache-2.0 (permissive) · e580a410194b1778 · report
get_gym_env gym_tensorflow/atari/tf_atari.py unverified Apache-2.0 (permissive) · eac41fa5ed269e17 · report
mean in 84 papers tf_util.py unverified Apache-2.0 (permissive) here; the stored copy is pointer only · 3dc7f82ede7de679 · report
sum in 11 papers tf_util.py unverified Apache-2.0 (permissive) here; the stored copy is pointer only · ec87a50ac64b9e20 · report
var in 16 papers tf_util.py unverified Apache-2.0 (permissive) here; the stored copy is pointer only · 9071c058ed8fc811 · report
wrap_deepmind in 17 papers atari_wrappers.py unverified Apache-2.0 (permissive) · a8c33fc375f8db7e · report

Tasks

Atari GamesDeep Reinforcement LearningReinforcement LearningReinforcement Learning (RL)reinforcement-learning

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Atari Games Atari 2600 Alien Ape-X Score 40804.9 #8 of 49 Archive leaderboard report
Atari Games Atari 2600 Amidar Ape-X Score 8659.2 #4 of 48 Archive leaderboard report
Atari Games Atari 2600 Assault Ape-X Score 24559.4 #8 of 45 Archive leaderboard report
Atari Games Atari 2600 Asterix Ape-X Score 313305 #15 of 49 Archive leaderboard report
Atari Games Atari 2600 Asteroids Ape-X Score 155495.1 #7 of 43 Archive leaderboard report
Atari Games Atari 2600 Atlantis Ape-X Score 944497.5 #17 of 42 Archive leaderboard report
Atari Games Atari 2600 Bank Heist Ape-X Score 1716.4 #4 of 45 Archive leaderboard report
Atari Games Atari 2600 Battle Zone Ape-X Score 98895 #7 of 47 Archive leaderboard report
Atari Games Atari 2600 Beam Rider Ape-X Score 63305.2 #8 of 49 Archive leaderboard report
Atari Games Atari 2600 Berzerk Ape-X Score 57196.7 #4 of 39 Archive leaderboard report
Atari Games Atari 2600 Bowling Ape-X Score 17.6 #44 of 44 Archive leaderboard report
Atari Games Atari 2600 Boxing Ape-X Score 100 #2 of 45 Archive leaderboard report
Atari Games Atari 2600 Breakout Ape-X Score 800.9 #8 of 58 Archive leaderboard report
Atari Games Atari 2600 Centipede Ape-X Score 12974 #14 of 45 Archive leaderboard report
Atari Games Atari 2600 Chopper Command Ape-X Score 721851 #8 of 45 Archive leaderboard report
Atari Games Atari 2600 Crazy Climber Ape-X Score 320426 #4 of 49 Archive leaderboard report
Atari Games Atari 2600 Defender Ape-X Score 411943.5 #9 of 21 Archive leaderboard report
Atari Games Atari 2600 Demon Attack Ape-X Score 133086.4 #9 of 46 Archive leaderboard report
Atari Games Atari 2600 Double Dunk Ape-X Score 23.5 #9 of 43 Archive leaderboard report
Atari Games Atari 2600 Enduro Ape-X Score 2177.4 #15 of 48 Archive leaderboard report
Atari Games Atari 2600 Fishing Derby Ape-X Score 44.4 #16 of 44 Archive leaderboard report
Atari Games Atari 2600 Freeway Ape-X Score 33.7 #15 of 59 Archive leaderboard report
Atari Games Atari 2600 Frostbite Ape-X Score 9328.6 #11 of 53 Archive leaderboard report
Atari Games Atari 2600 Gopher Ape-X Score 120500.9 #6 of 43 Archive leaderboard report
Atari Games Atari 2600 Gravitar Ape-X Score 1598.5 #19 of 53 Archive leaderboard report
Atari Games Atari 2600 HERO Ape-X Score 31655.9 #12 of 44 Archive leaderboard report
Atari Games Atari 2600 Ice Hockey Ape-X Score 33 #9 of 43 Archive leaderboard report
Atari Games Atari 2600 James Bond Ape-X Score 21322.5 #11 of 45 Archive leaderboard report
Atari Games Atari 2600 Kangaroo Ape-X Score 1416 #38 of 47 Archive leaderboard report
Atari Games Atari 2600 Krull Ape-X Score 11741.4 #10 of 45 Archive leaderboard report
Atari Games Atari 2600 Kung-Fu Master Ape-X Score 97829.5 #10 of 43 Archive leaderboard report
Atari Games Atari 2600 Montezuma's Revenge Ape-X Score 2500.0 #15 of 50 Archive leaderboard report
Atari Games Atari 2600 Ms. Pacman Ape-X Score 11255.2 #9 of 47 Archive leaderboard report
Atari Games Atari 2600 Name This Game Ape-X Score 25783.3 #8 of 43 Archive leaderboard report
Atari Games Atari 2600 Phoenix Ape-X Score 224491.1 #9 of 21 Archive leaderboard report
Atari Games Atari 2600 Pitfall! Ape-X Score -0.6 #21 of 23 Archive leaderboard report
Atari Games Atari 2600 Pong Ape-X Score 20.9 #21 of 52 Archive leaderboard report
Atari Games Atari 2600 Private Eye Ape-X Score 49.8 #51 of 52 Archive leaderboard report
Atari Games Atari 2600 Q*Bert Ape-X Score 302391.3 #5 of 57 Archive leaderboard report
Atari Games Atari 2600 River Raid Ape-X Score 63864.4 #3 of 41 Archive leaderboard report
Atari Games Atari 2600 Road Runner Ape-X Score 222234.5 #9 of 44 Archive leaderboard report
Atari Games Atari 2600 Robotank Ape-X Score 73.8 #10 of 42 Archive leaderboard report
Atari Games Atari 2600 Seaquest Ape-X Score 392952.3 #9 of 57 Archive leaderboard report
Atari Games Atari 2600 Skiing Ape-X Score -10789.9 #11 of 23 Archive leaderboard report
Atari Games Atari 2600 Solaris Ape-X Score 2892.9 #18 of 23 Archive leaderboard report
Atari Games Atari 2600 Space Invaders Ape-X Score 54681 #6 of 55 Archive leaderboard report
Atari Games Atari 2600 Star Gunner Ape-X Score 434342.5 #7 of 43 Archive leaderboard report
Atari Games Atari 2600 Surround Ape-X Score 7.1 #9 of 15 Archive leaderboard report
Atari Games Atari 2600 Tennis Ape-X Score 23.9 #4 of 43 Archive leaderboard report
Atari Games Atari 2600 Time Pilot Ape-X Score 87085 #8 of 44 Archive leaderboard report
Atari Games Atari 2600 Tutankham Ape-X Score 272.6 #13 of 44 Archive leaderboard report
Atari Games Atari 2600 Up and Down Ape-X Score 401884.3 #9 of 44 Archive leaderboard report
Atari Games Atari 2600 Venture Ape-X Score 1813 #10 of 55 Archive leaderboard report
Atari Games Atari 2600 Video Pinball Ape-X Score 565163.2 #14 of 42 Archive leaderboard report
Atari Games Atari 2600 Wizard of Wor Ape-X Score 46204 #8 of 41 Archive leaderboard report
Atari Games Atari 2600 Yars Revenge Ape-X Score 148594.8 #9 of 17 Archive leaderboard report
Atari Games Atari 2600 Zaxxon Ape-X Score 42285.5 #8 of 41 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Introduced by this paper: Ape-X, Ape-X DPG, Ape-X DQN

AdamApe-XApe-X DPGApe-X DQNBatch NormalizationConvolutionDDPGDense ConnectionsDouble Q-learningDueling NetworkExperience ReplayN-step ReturnsPrioritized Experience ReplayRMSPropReLUWeight Decay

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections