{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/torchbeast-a-pytorch-platform-for-distributed","title":"TorchBeast: A PyTorch Platform for Distributed RL","arxiv_id":"1910.03552","date":"2019-10-08","proceeding":null,"authors":["Heinrich Küttler","Nantas Nardelli","Thibaut Lavril","Marco Selvatici","Viswanath Sivakumar","Tim Rocktäschel","Edward Grefenstette"],"abstract":"TorchBeast is a platform for reinforcement learning (RL) research in PyTorch. It implements a version of the popular IMPALA algorithm for fast, asynchronous, parallel training of RL agents. Additionally, TorchBeast has simplicity as an explicit design goal: We provide both a pure-Python implementation (\"MonoBeast\") as well as a multi-machine high-performance version (\"PolyBeast\"). In the latter, parts of the implementation are written in C++, but all parts pertaining to machine learning are kept in simple Python using PyTorch, with the environments provided using the OpenAI Gym interface. This enables researchers to conduct scalable RL research using TorchBeast without any programming knowledge beyond Python and PyTorch. In this paper, we describe the TorchBeast design principles and implementation and demonstrate that it performs on-par with IMPALA on Atari. TorchBeast is released as an open-source package under the Apache 2.0 license and is available at \\url{https://github.com/facebookresearch/torchbeast}.","url_abs":"https://arxiv.org/abs/1910.03552v1","url_pdf":"https://arxiv.org/pdf/1910.03552v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"torchbeast-a-pytorch-platform-for-distributed","repo_url":"https://github.com/heiner/scalable_agent","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"torchbeast-a-pytorch-platform-for-distributed","repo_url":"https://github.com/facebookresearch/mvfst-rl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"torchbeast-a-pytorch-platform-for-distributed","repo_url":"https://github.com/facebookresearch/torchbeast","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"openai-gym","task_name":"OpenAI Gym"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"entropy-regularization","method_name":"Entropy Regularization"},{"method_slug":"experience-replay","method_name":"Experience Replay"},{"method_slug":"gradient-clipping","method_name":"Gradient Clipping"},{"method_slug":"impala","method_name":"IMPALA"},{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"rmsprop","method_name":"RMSProp"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"},{"method_slug":"torchbeast","method_name":"TorchBeast"},{"method_slug":"v-trace","method_name":"V-trace"}],"datasets_introduced":[],"methods_introduced":[{"slug":"torchbeast","name":"TorchBeast","full_name":"TorchBeast"}],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1910.03552","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1910.03552"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/mvfst-rl","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/torchbeast","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/heiner/scalable_agent","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":3,"ran_fixture":1,"unverified":5},"by_repo_kind":{"official":{"samples":6,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"e9123707cba84b2c","entry":"compute_baseline_loss","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"e9123707cba84b2c"}},{"code_sha256_prefix":"a27a9dc97eefda7f","entry":"compute_entropy_loss","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"a27a9dc97eefda7f"}},{"code_sha256_prefix":"d6d7e0906f1c57b2","entry":"compute_policy_gradient_loss","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"d6d7e0906f1c57b2"}},{"code_sha256_prefix":"a75de4dc4c61d7dc","entry":"log_probs_from_logits_and_actions","repo":"heiner/scalable_agent","repo_kind":"official","path":"vtrace.py","file_url":"https://github.com/heiner/scalable_agent/blob/HEAD/vtrace.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a75de4dc4c61d7dc"}},{"code_sha256_prefix":"777b5a676e76d593","entry":"compute_baseline_loss","repo":"heiner/scalable_agent","repo_kind":"official","path":"experiment.py","file_url":"https://github.com/heiner/scalable_agent/blob/HEAD/experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"777b5a676e76d593"}},{"code_sha256_prefix":"50c20c55f1b07c8e","entry":"compute_entropy_loss","repo":"heiner/scalable_agent","repo_kind":"official","path":"experiment.py","file_url":"https://github.com/heiner/scalable_agent/blob/HEAD/experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"50c20c55f1b07c8e"}},{"code_sha256_prefix":"be48a80c545e0430","entry":"compute_human_normalized_score","repo":"heiner/scalable_agent","repo_kind":"official","path":"dmlab30.py","file_url":"https://github.com/heiner/scalable_agent/blob/HEAD/dmlab30.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"be48a80c545e0430"}},{"code_sha256_prefix":"e11ee068514828e6","entry":"from_importance_weights","repo":"heiner/scalable_agent","repo_kind":"official","path":"vtrace.py","file_url":"https://github.com/heiner/scalable_agent/blob/HEAD/vtrace.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e11ee068514828e6"}},{"code_sha256_prefix":"1175c77e601077b8","entry":"from_logits","repo":"heiner/scalable_agent","repo_kind":"official","path":"vtrace.py","file_url":"https://github.com/heiner/scalable_agent/blob/HEAD/vtrace.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1175c77e601077b8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}