{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-reactor-a-fast-and-sample-efficient-actor","title":"The Reactor: A fast and sample-efficient Actor-Critic agent for Reinforcement Learning","arxiv_id":"1704.04651","date":"2017-04-15","proceeding":"ICLR 2018 1","authors":["Audrunas Gruslys","Will Dabney","Mohammad Gheshlaghi Azar","Bilal Piot","Marc Bellemare","Remi Munos"],"abstract":"In this work we present a new agent architecture, called Reactor, which\ncombines multiple algorithmic and architectural contributions to produce an\nagent with higher sample-efficiency than Prioritized Dueling DQN (Wang et al.,\n2016) and Categorical DQN (Bellemare et al., 2017), while giving better\nrun-time performance than A3C (Mnih et al., 2016). Our first contribution is a\nnew policy evaluation algorithm called Distributional Retrace, which brings\nmulti-step off-policy updates to the distributional reinforcement learning\nsetting. The same approach can be used to convert several classes of multi-step\npolicy evaluation algorithms designed for expected value evaluation into\ndistributional ones. Next, we introduce the \\b{eta}-leave-one-out policy\ngradient algorithm which improves the trade-off between variance and bias by\nusing action values as a baseline. Our final algorithmic contribution is a new\nprioritized replay algorithm for sequences, which exploits the temporal\nlocality of neighboring observations for more efficient replay prioritization.\nUsing the Atari 2600 benchmarks, we show that each of these innovations\ncontribute to both the sample efficiency and final agent performance. Finally,\nwe demonstrate that Reactor reaches state-of-the-art performance after 200\nmillion frames and less than a day of training.","url_abs":"http://arxiv.org/abs/1704.04651v2","url_pdf":"http://arxiv.org/pdf/1704.04651v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"atari-games","task_name":"Atari Games"},{"task_slug":"distributional-reinforcement-learning","task_name":"Distributional Reinforcement Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[{"method_slug":"a3c","method_name":"A3C"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dqn","method_name":"DQN"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"entropy-regularization","method_name":"Entropy Regularization"},{"method_slug":"q-learning","method_name":"Q-Learning"},{"method_slug":"retrace","method_name":"Retrace"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/atari-games-on-atari-2600-alien","task":"Atari Games","dataset":"Atari 2600 Alien","model":"Reactor 500M","rank_in_archive_order":11,"of":49,"metrics":{"Score":"12689.1"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-amidar","task":"Atari Games","dataset":"Atari 2600 Amidar","model":"Reactor 500M","rank_in_archive_order":25,"of":48,"metrics":{"Score":"1015.8"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-assault","task":"Atari Games","dataset":"Atari 2600 Assault","model":"Reactor 500M","rank_in_archive_order":20,"of":45,"metrics":{"Score":"8323.3"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-asterix","task":"Atari Games","dataset":"Atari 2600 Asterix","model":"Reactor 500M","rank_in_archive_order":19,"of":49,"metrics":{"Score":"205914.0"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-asteroids","task":"Atari Games","dataset":"Atari 2600 Asteroids","model":"Reactor 500M","rank_in_archive_order":18,"of":43,"metrics":{"Score":"3726.1"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-atlantis","task":"Atari Games","dataset":"Atari 2600 Atlantis","model":"Reactor 500M","rank_in_archive_order":34,"of":42,"metrics":{"Score":"302831.0"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-bank-heist","task":"Atari Games","dataset":"Atari 2600 Bank Heist","model":"Reactor 500M","rank_in_archive_order":14,"of":45,"metrics":{"Score":"1259.7"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-battle-zone","task":"Atari Games","dataset":"Atari 2600 Battle Zone","model":"Reactor 500M","rank_in_archive_order":11,"of":47,"metrics":{"Score":"64070.0"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-beam-rider","task":"Atari Games","dataset":"Atari 2600 Beam Rider","model":"Reactor 500M","rank_in_archive_order":31,"of":49,"metrics":{"Score":"11033.4"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-berzerk","task":"Atari Games","dataset":"Atari 2600 Berzerk","model":"Reactor 500M","rank_in_archive_order":15,"of":39,"metrics":{"Score":"2303.1"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-bowling","task":"Atari Games","dataset":"Atari 2600 Bowling","model":"Reactor 500M","rank_in_archive_order":16,"of":44,"metrics":{"Score":"81.0"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-boxing","task":"Atari Games","dataset":"Atari 2600 Boxing","model":"Reactor 500M","rank_in_archive_order":17,"of":45,"metrics":{"Score":"99.4"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-breakout","task":"Atari Games","dataset":"Atari 2600 Breakout","model":"Reactor 500M","rank_in_archive_order":20,"of":58,"metrics":{"Score":"514.8"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-centipede","task":"Atari Games","dataset":"Atari 2600 Centipede","model":"Reactor 500M","rank_in_archive_order":42,"of":45,"metrics":{"Score":"3422.0"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-chopper-command","task":"Atari Games","dataset":"Atari 2600 Chopper Command","model":"Reactor 500M","rank_in_archive_order":9,"of":45,"metrics":{"Score":"107779.0"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-crazy-climber","task":"Atari Games","dataset":"Atari 2600 Crazy Climber","model":"Reactor 500M","rank_in_archive_order":6,"of":49,"metrics":{"Score":"236422.0"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-defender","task":"Atari Games","dataset":"Atari 2600 Defender","model":"Reactor 500M","rank_in_archive_order":10,"of":21,"metrics":{"Score":"223025.0"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-demon-attack","task":"Atari Games","dataset":"Atari 2600 Demon Attack","model":"Reactor 500M","rank_in_archive_order":16,"of":46,"metrics":{"Score":"115154.0"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-double-dunk","task":"Atari Games","dataset":"Atari 2600 Double Dunk","model":"Reactor 500M","rank_in_archive_order":10,"of":43,"metrics":{"Score":"23.0"},"uses_additional_data":false},{"leaderboard":"/sota/atari-games-on-atari-2600-enduro","task":"Atari Games","dataset":"Atari 2600 Enduro","model":"Reactor 500M","rank_in_archive_order":13,"of":48,"metrics":{"Score":"2224.2"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1704.04651","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}