{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deployable-reinforcement-learning-with","title":"Deployable Reinforcement Learning with Variable Control Rate","arxiv_id":"2401.09286","date":"2024-01-17","proceeding":null,"authors":["Dong Wang","Giovanni Beltrame"],"abstract":"Deploying controllers trained with Reinforcement Learning (RL) on real robots can be challenging: RL relies on agents' policies being modeled as Markov Decision Processes (MDPs), which assume an inherently discrete passage of time. The use of MDPs results in that nearly all RL-based control systems employ a fixed-rate control strategy with a period (or time step) typically chosen based on the developer's experience or specific characteristics of the application environment. Unfortunately, the system should be controlled at the highest, worst-case frequency to ensure stability, which can demand significant computational and energy resources and hinder the deployability of the controller on onboard hardware. Adhering to the principles of reactive programming, we surmise that applying control actions only when necessary enables the use of simpler hardware and helps reduce energy consumption. We challenge the fixed frequency assumption by proposing a variant of RL with variable control rate. In this approach, the policy decides the action the agent should take as well as the duration of the time step associated with that action. In our new setting, we expand Soft Actor-Critic (SAC) to compute the optimal policy with a variable control rate, introducing the Soft Elastic Actor-Critic (SEAC) algorithm. We show the efficacy of SEAC through a proof-of-concept simulation driving an agent with Newtonian kinematics. Our experiments show higher average returns, shorter task completion times, and reduced computational resources when compared to fixed rate policies.","url_abs":"https://arxiv.org/abs/2401.09286v2","url_pdf":"https://arxiv.org/pdf/2401.09286v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deployable-reinforcement-learning-with","repo_url":"https://github.com/alpaficia/SEAC_Pytorch_release","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.09286","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.09286"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/alpaficia/SEAC_Pytorch_release","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"dbede9400bf58c5b","entry":"Action_adapter","repo":"alpaficia/SEAC_Pytorch_release","repo_kind":"official","path":"Adapter.py","file_url":"https://github.com/alpaficia/SEAC_Pytorch_release/blob/HEAD/Adapter.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dbede9400bf58c5b"}},{"code_sha256_prefix":"4efe181a3a3fa4ec","entry":"Action_t_relu6_adapter","repo":"alpaficia/SEAC_Pytorch_release","repo_kind":"official","path":"Adapter.py","file_url":"https://github.com/alpaficia/SEAC_Pytorch_release/blob/HEAD/Adapter.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4efe181a3a3fa4ec"}},{"code_sha256_prefix":"7fde9410d5f59811","entry":"Action_t_relu6_adapter_reverse","repo":"alpaficia/SEAC_Pytorch_release","repo_kind":"official","path":"Adapter.py","file_url":"https://github.com/alpaficia/SEAC_Pytorch_release/blob/HEAD/Adapter.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7fde9410d5f59811"}},{"code_sha256_prefix":"9b95be60df57db51","entry":"build_net","repo":"alpaficia/SEAC_Pytorch_release","repo_kind":"official","path":"SEAC.py","file_url":"https://github.com/alpaficia/SEAC_Pytorch_release/blob/HEAD/SEAC.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9b95be60df57db51"}},{"code_sha256_prefix":"c2c1aae2bf247d8d","entry":"coordinate_system_conversion","repo":"alpaficia/SEAC_Pytorch_release","repo_kind":"official","path":"envs/dynamic_world.py","file_url":"https://github.com/alpaficia/SEAC_Pytorch_release/blob/HEAD/envs/dynamic_world.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c2c1aae2bf247d8d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}