{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/envpool-a-highly-parallel-reinforcement","title":"EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Engine","arxiv_id":"2206.10558","date":"2022-06-21","proceeding":null,"authors":["Jiayi Weng","Min Lin","Shengyi Huang","Bo Liu","Denys Makoviichuk","Viktor Makoviychuk","Zichen Liu","Yufan Song","Ting Luo","Yukun Jiang","Zhongwen Xu","Shuicheng Yan"],"abstract":"There has been significant progress in developing reinforcement learning (RL) training systems. Past works such as IMPALA, Apex, Seed RL, Sample Factory, and others, aim to improve the system's overall throughput. In this paper, we aim to address a common bottleneck in the RL training system, i.e., parallel environment execution, which is often the slowest part of the whole system but receives little attention. With a curated design for paralleling RL environments, we have improved the RL environment simulation speed across different hardware setups, ranging from a laptop and a modest workstation, to a high-end machine such as NVIDIA DGX-A100. On a high-end machine, EnvPool achieves one million frames per second for the environment execution on Atari environments and three million frames per second on MuJoCo environments. When running EnvPool on a laptop, the speed is 2.8x that of the Python subprocess. Moreover, great compatibility with existing RL training libraries has been demonstrated in the open-sourced community, including CleanRL, rl_games, DeepMind Acme, etc. Finally, EnvPool allows researchers to iterate their ideas at a much faster pace and has great potential to become the de facto RL environment execution engine. Example runs show that it only takes five minutes to train agents to play Atari Pong and MuJoCo Ant on a laptop. EnvPool is open-sourced at https://github.com/sail-sg/envpool.","url_abs":"https://arxiv.org/abs/2206.10558v2","url_pdf":"https://arxiv.org/pdf/2206.10558v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"envpool-a-highly-parallel-reinforcement","repo_url":"https://github.com/sail-sg/envpool","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"envpool-a-highly-parallel-reinforcement","repo_url":"https://github.com/vwxyzjn/envpool-cleanrl","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"envpool-a-highly-parallel-reinforcement","repo_url":"https://github.com/vwxyzjn/cleanrl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"mujoco","task_name":"MuJoCo"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"entropy-regularization","method_name":"Entropy Regularization"},{"method_slug":"experience-replay","method_name":"Experience Replay"},{"method_slug":"gradient-clipping","method_name":"Gradient Clipping"},{"method_slug":"impala","method_name":"IMPALA"},{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"ppo","method_name":"PPO"},{"method_slug":"rmsprop","method_name":"RMSProp"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"speed","method_name":"SPEED"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"},{"method_slug":"v-trace","method_name":"V-trace"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2206.10558","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.10558"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/sail-sg/envpool","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vwxyzjn/cleanrl","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vwxyzjn/envpool-cleanrl","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"unverified":4},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1},"named_in_paper":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"beb42d440ccf2925","entry":"layer_init","repo":"vwxyzjn/envpool-cleanrl","repo_kind":"named_in_paper","path":"ppo_atari_envpool.py","file_url":"https://github.com/vwxyzjn/envpool-cleanrl/blob/HEAD/ppo_atari_envpool.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"beb42d440ccf2925"}},{"code_sha256_prefix":"f5ee14adedd4aa20","entry":"asset_base_path","repo":"sail-sg/envpool","repo_kind":"official","path":"envpool/registration.py","file_url":"https://github.com/sail-sg/envpool/blob/HEAD/envpool/registration.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f5ee14adedd4aa20"}},{"code_sha256_prefix":"055188fa794e90ef","entry":"encode","repo":"sail-sg/envpool","repo_kind":"official","path":"envpool/craftax/oracle.py","file_url":"https://github.com/sail-sg/envpool/blob/HEAD/envpool/craftax/oracle.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"055188fa794e90ef"}},{"code_sha256_prefix":"29f63787cb6f151f","entry":"flatten","repo":"sail-sg/envpool","repo_kind":"official","path":"envpool/craftax/oracle.py","file_url":"https://github.com/sail-sg/envpool/blob/HEAD/envpool/craftax/oracle.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"29f63787cb6f151f"}},{"code_sha256_prefix":"3ee033959764bfd6","entry":"make_env","repo":"vwxyzjn/envpool-cleanrl","repo_kind":"named_in_paper","path":"ppo_continuous_action.py","file_url":"https://github.com/vwxyzjn/envpool-cleanrl/blob/HEAD/ppo_continuous_action.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3ee033959764bfd6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}