{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/overcoming-slow-decision-frequencies-in","title":"Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control","arxiv_id":"2410.08979","date":"2024-10-11","proceeding":null,"authors":["Devdhar Patel","Hava Siegelmann"],"abstract":"Reinforcement learning (RL) is rapidly reaching and surpassing human-level control capabilities. However, state-of-the-art RL algorithms often require timesteps and reaction times significantly faster than human capabilities, which is impractical in real-world settings and typically necessitates specialized hardware. Such speeds are difficult to achieve in the real world and often requires specialized hardware. We introduce Sequence Reinforcement Learning (SRL), an RL algorithm designed to produce a sequence of actions for a given input state, enabling effective control at lower decision frequencies. SRL addresses the challenges of learning action sequences by employing both a model and an actor-critic architecture operating at different temporal scales. We propose a \"temporal recall\" mechanism, where the critic uses the model to estimate intermediate states between primitive actions, providing a learning signal for each individual action within the sequence. Once training is complete, the actor can generate action sequences independently of the model, achieving model-free control at a slower frequency. We evaluate SRL on a suite of continuous control tasks, demonstrating that it achieves performance comparable to state-of-the-art algorithms while significantly reducing actor sample complexity. To better assess performance across varying decision frequencies, we introduce the Frequency-Averaged Score (FAS) metric. Our results show that SRL significantly outperforms traditional RL algorithms in terms of FAS, making it particularly suitable for applications requiring variable decision frequencies. Additionally, we compare SRL with model-based online planning, showing that SRL achieves superior FAS while leveraging the same model during training that online planners use for planning.","url_abs":"https://arxiv.org/abs/2410.08979v2","url_pdf":"https://arxiv.org/pdf/2410.08979v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"overcoming-slow-decision-frequencies-in","repo_url":"https://github.com/dee0512/Temporally-Layered-Architecture","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"continuous-control","task_name":"continuous-control"},{"task_slug":"model","task_name":"model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.08979","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.08979"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dee0512/Temporally-Layered-Architecture","reach":null}],"summary":{"ran":4,"ran_honours":1,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1},"community":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"464ed648e3d7a3e2","entry":"Actor","repo":"dee0512/Temporally-Layered-Architecture","repo_kind":"official","path":"model.py","file_url":"https://github.com/dee0512/Temporally-Layered-Architecture/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"464ed648e3d7a3e2"}},{"code_sha256_prefix":"0a41250eec295ce6","entry":"Critic","repo":"dee0512/Temporally-Layered-Architecture","repo_kind":"official","path":"model.py","file_url":"https://github.com/dee0512/Temporally-Layered-Architecture/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"0a41250eec295ce6"}},{"code_sha256_prefix":"4cfa4869be2885b7","entry":"Q","repo":"dee0512/Temporally-Layered-Architecture","repo_kind":"official","path":"model.py","file_url":"https://github.com/dee0512/Temporally-Layered-Architecture/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"4cfa4869be2885b7"}},{"code_sha256_prefix":"a965e50dc543532e","entry":"TD3","repo":"dee0512/Temporally-Layered-Architecture","repo_kind":"official","path":"model.py","file_url":"https://github.com/dee0512/Temporally-Layered-Architecture/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"a965e50dc543532e"}},{"code_sha256_prefix":"54f3f957abc96d34","entry":"evaluate_policy","repo":"dee0512/sequence-reinforcement-learning","repo_kind":"community","path":"scripts/eval_sac.py","file_url":"https://github.com/dee0512/sequence-reinforcement-learning/blob/HEAD/scripts/eval_sac.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"54f3f957abc96d34"}},{"code_sha256_prefix":"d64a2fb8d7232eee","entry":"TLA","repo":"dee0512/Temporally-Layered-Architecture","repo_kind":"official","path":"model.py","file_url":"https://github.com/dee0512/Temporally-Layered-Architecture/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"d64a2fb8d7232eee"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}