{"url":"/method/ape-x","slug":"ape-x","name":"Ape-X","full_name":"Ape-X","full_name_withheld":false,"description_markdown":"**Ape-X** is a distributed architecture for deep reinforcement learning. The algorithm decouples acting from learning: the actors interact with their own instances of the environment by selecting actions according to a shared neural network, and accumulate the resulting experience in a shared [experience replay](https://paperswithcode.com/method/experience-replay) memory; the learner replays samples of experience and updates the neural network. The architecture relies on [prioritized experience replay](https://paperswithcode.com/method/prioritized-experience-replay) to focus only on the most significant data generated by the actors.\r\n\r\nIn contrast to Gorila, Ape-X uses a shared, centralized replay memory, and instead of sampling\r\nuniformly, it prioritizes, to sample the most useful data more often. All communications are batched with the centralized replay, increasing the efficiency and throughput at the cost of some latency. \r\nAnd by learning off-policy, Ape-X has the ability to combine data from many distributed actors, by giving the different actors different exploration policies, broadening the diversity of the experience they jointly encounter.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Distributed Prioritized Experience Replay","paper":"/paper/distributed-prioritized-experience-replay","first_author":"Dan Horgan","n_authors":7,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/distributed-prioritized-experience-replay"},"source":{"url":"http://arxiv.org/abs/1803.00933v1","title":"Distributed Prioritized Experience Replay","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Reinforcement Learning","area_id":"reinforcement-learning","collection":"Distributed Reinforcement Learning","url":"/methods/category/distributed-reinforcement-learning","pwc_aliases":[]}],"n_papers_tagged":10,"archive_num_papers":10,"papers_newest_first":[{"paper":null,"title":"Safety Margins for Reinforcement Learning","date":"2023-07-25","arxiv_id":"2307.13642","n_code_links":0,"syntology":null},{"paper":null,"title":"Parallel Reinforcement Learning Simulation for Visual Quadrotor Navigation","date":"2022-09-22","arxiv_id":"2209.11094","n_code_links":0,"syntology":null},{"paper":"/paper/a-coevolutionairy-approach-to-deep-multi","title":"A coevolutionary approach to deep multi-agent reinforcement learning","date":"2021-04-12","arxiv_id":"2104.05610","n_code_links":1,"syntology":null},{"paper":null,"title":"A Learning Approach to Robot-Agnostic Force-Guided High Precision Assembly","date":"2020-10-15","arxiv_id":"2010.08052","n_code_links":0,"syntology":null},{"paper":null,"title":"Dynamic Experience Replay","date":"2020-03-04","arxiv_id":"2003.02372","n_code_links":0,"syntology":null},{"paper":"/paper/google-research-football-a-novel","title":"Google Research Football: A Novel Reinforcement Learning Environment","date":"2019-07-25","arxiv_id":"1907.11180","n_code_links":1,"syntology":{"ran":0,"of":4,"unverified":4,"pointer_only":0}},{"paper":"/paper/macro-action-selection-with-deep","title":"Macro action selection with deep reinforcement learning in StarCraft","date":"2018-12-02","arxiv_id":"1812.00336","n_code_links":1,"syntology":null},{"paper":"/paper/an-intriguing-failing-of-convolutional-neural","title":"An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution","date":"2018-07-09","arxiv_id":"1807.03247","n_code_links":24,"syntology":{"ran":4,"of":5,"unverified":1,"pointer_only":0}},{"paper":null,"title":"Deep Curiosity Search: Intra-Life Exploration Can Improve Performance on Challenging Deep Reinforcement Learning Problems","date":"2018-06-01","arxiv_id":"1806.00553","n_code_links":0,"syntology":null},{"paper":"/paper/distributed-prioritized-experience-replay","title":"Distributed Prioritized Experience Replay","date":"2018-03-02","arxiv_id":"1803.00933","n_code_links":15,"syntology":{"ran":0,"of":15,"unverified":15,"pointer_only":0}}],"papers_shown":10,"tasks":[{"task":"/task/reinforcement-learning","name":"Reinforcement Learning","papers":8},{"task":"/task/reinforcement-learning-1","name":"Reinforcement Learning (RL)","papers":6},{"task":"/task/reinforcement-learning-2","name":"reinforcement-learning","papers":6},{"task":"/task/atari-games","name":"Atari Games","papers":3},{"task":"/task/deep-reinforcement-learning","name":"Deep Reinforcement Learning","papers":3},{"task":"/task/decision-making","name":"Decision Making","papers":1},{"task":"/task/game-of-football","name":"Game of Football","papers":1},{"task":"/task/image-classification","name":"Image Classification","papers":1},{"task":"/task/montezumas-revenge","name":"Montezuma's Revenge","papers":1},{"task":"/task/multi-agent-reinforcement-learning","name":"Multi-agent Reinforcement Learning","papers":1},{"task":"/task/navigate","name":"Navigate","papers":1},{"task":"/task/openai-gym","name":"OpenAI Gym","papers":1},{"task":"/task/real-time-strategy-games","name":"Real-Time Strategy Games","papers":1},{"task":"/task/starcraft","name":"Starcraft","papers":1},{"task":"/task/translation","name":"Translation","papers":1},{"task":"/task/high","name":"Vocal Bursts Intensity Prediction","papers":1}],"tasks_shown":16,"n_tasks":16,"usage_by_year":[{"year":"2018","papers":4},{"year":"2019","papers":1},{"year":"2020","papers":2},{"year":"2021","papers":1},{"year":"2022","papers":1},{"year":"2023","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/ape-x"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}