{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/nappo-modular-and-scalable-reinforcement","title":"Integrating Distributed Architectures in Highly Modular RL Libraries","arxiv_id":"2007.02622","date":"2020-07-06","proceeding":null,"authors":["Albert Bou","Sebastian Dittert","Gianni de Fabritiis"],"abstract":"Advancing reinforcement learning (RL) requires tools that are flexible enough to easily prototype new methods while avoiding impractically slow experimental turnaround times. To match the first requirement, the most popular RL libraries advocate for highly modular agent composability, which facilitates experimentation and development. To solve challenging environments within reasonable time frames, scaling RL to large sampling and computing resources has proved a successful strategy. However, this capability has been so far difficult to combine with modularity. In this work, we explore design choices to allow agent composability both at a local and distributed level of execution. We propose a versatile approach that allows the definition of RL agents at different scales through independent reusable components. We demonstrate experimentally that our design choices allow us to reproduce classical benchmarks, explore multiple distributed architectures, and solve novel and complex environments while giving full control to the user in the agent definition and training scheme definition. We believe this work can provide useful insights to the next generation of RL libraries.","url_abs":"https://arxiv.org/abs/2007.02622v3","url_pdf":"https://arxiv.org/pdf/2007.02622v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"nappo-modular-and-scalable-reinforcement","repo_url":"https://github.com/PyTorchRL/pytorchrl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2007.02622","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2007.02622"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PyTorchRL/pytorchrl","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":8},"by_repo_kind":{"listed":{"samples":8,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9c88cb72fa692623","entry":"average_gradients","repo":"PyTorchRL/pytorchrl","repo_kind":"listed","path":"pytorchrl/scheme/utils.py","file_url":"https://github.com/PyTorchRL/pytorchrl/blob/HEAD/pytorchrl/scheme/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9c88cb72fa692623"}},{"code_sha256_prefix":"8a172de9485e8562","entry":"bt","repo":"PyTorchRL/pytorchrl","repo_kind":"listed","path":"pytorchrl/agent/algorithms/utils.py","file_url":"https://github.com/PyTorchRL/pytorchrl/blob/HEAD/pytorchrl/agent/algorithms/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a172de9485e8562"}},{"code_sha256_prefix":"72a80ce7592e38cd","entry":"btr","repo":"PyTorchRL/pytorchrl","repo_kind":"listed","path":"pytorchrl/agent/algorithms/utils.py","file_url":"https://github.com/PyTorchRL/pytorchrl/blob/HEAD/pytorchrl/agent/algorithms/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"72a80ce7592e38cd"}},{"code_sha256_prefix":"889e1ecd97fa6cbd","entry":"check_message","repo":"PyTorchRL/pytorchrl","repo_kind":"listed","path":"pytorchrl/scheme/utils.py","file_url":"https://github.com/PyTorchRL/pytorchrl/blob/HEAD/pytorchrl/scheme/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"889e1ecd97fa6cbd"}},{"code_sha256_prefix":"c668f382578e3aa4","entry":"clip_grad_norm_","repo":"PyTorchRL/pytorchrl","repo_kind":"listed","path":"pytorchrl/utils.py","file_url":"https://github.com/PyTorchRL/pytorchrl/blob/HEAD/pytorchrl/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c668f382578e3aa4"}},{"code_sha256_prefix":"5ec30e1da221a76e","entry":"gaussian_kl","repo":"PyTorchRL/pytorchrl","repo_kind":"listed","path":"pytorchrl/agent/algorithms/utils.py","file_url":"https://github.com/PyTorchRL/pytorchrl/blob/HEAD/pytorchrl/agent/algorithms/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5ec30e1da221a76e"}},{"code_sha256_prefix":"885df13bf6c29027","entry":"get_algorithm","repo":"PyTorchRL/pytorchrl","repo_kind":"listed","path":"pytorchrl/trainer.py","file_url":"https://github.com/PyTorchRL/pytorchrl/blob/HEAD/pytorchrl/trainer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"885df13bf6c29027"}},{"code_sha256_prefix":"b3a16c8114ffbb77","entry":"ray_get_and_free","repo":"PyTorchRL/pytorchrl","repo_kind":"listed","path":"pytorchrl/scheme/utils.py","file_url":"https://github.com/PyTorchRL/pytorchrl/blob/HEAD/pytorchrl/scheme/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b3a16c8114ffbb77"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}