{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tackling-unbounded-state-spaces-in-continuing","title":"Learning to Stabilize Online Reinforcement Learning in Unbounded State Spaces","arxiv_id":"2306.01896","date":"2023-06-02","proceeding":null,"authors":["Brahma S. Pavse","Matthew Zurek","Yudong Chen","Qiaomin Xie","Josiah P. Hanna"],"abstract":"In many reinforcement learning (RL) applications, we want policies that reach desired states and then keep the controlled system within an acceptable region around the desired states over an indefinite period of time. This latter objective is called stability and is especially important when the state space is unbounded, such that the states can be arbitrarily far from each other and the agent can drift far away from the desired states. For example, in stochastic queuing networks, where queues of waiting jobs can grow without bound, the desired state is all-zero queue lengths. Here, a stable policy ensures queue lengths are finite while an optimal policy minimizes queue lengths. Since an optimal policy is also stable, one would expect that RL algorithms would implicitly give us stable policies. However, in this work, we find that deep RL algorithms that directly minimize the distance to the desired state during online training often result in unstable policies, i.e., policies that drift far away from the desired state. We attribute this instability to poor credit-assignment for destabilizing actions. We then introduce an approach based on two ideas: 1) a Lyapunov-based cost-shaping technique and 2) state transformations to the unbounded state space. We conduct an empirical study on various queueing networks and traffic signal control problems and find that our approach performs competitively against strong baselines with knowledge of the transition dynamics. Our code is available here: https://github.com/Badger-RL/STOP.","url_abs":"https://arxiv.org/abs/2306.01896v3","url_pdf":"https://arxiv.org/pdf/2306.01896v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tackling-unbounded-state-spaces-in-continuing","repo_url":"https://github.com/badger-rl/stop","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"traffic-signal-control","task_name":"Traffic Signal Control"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2306.01896","atlas_url":"https://app.syntology.ai/?focus=2306.01896","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.01896"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Badger-RL/STOP","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/badger-rl/stop","reach":{"status":"ok"}}],"summary":{"ran":1,"ran_honours":2,"ran_violates":2,"ran_fixture":1},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"cbfacd34edc685bc","entry":"NModelNetwork","repo":"badger-rl/stop","repo_kind":"official","path":"nmodel.py","file_url":"https://github.com/badger-rl/stop/blob/HEAD/nmodel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cbfacd34edc685bc"}},{"code_sha256_prefix":"604c5823be725722","entry":"sigmoid","repo":"badger-rl/stop","repo_kind":"official","path":"nmodel.py","file_url":"https://github.com/badger-rl/stop/blob/HEAD/nmodel.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"604c5823be725722"}},{"code_sha256_prefix":"f017532fc389cbfe","entry":"str2bool","repo":"Badger-RL/STOP","repo_kind":"official","path":"plot.py","file_url":"https://github.com/Badger-RL/STOP/blob/HEAD/plot.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f017532fc389cbfe"}},{"code_sha256_prefix":"69b48660ab8e75d9","entry":"symlog","repo":"badger-rl/stop","repo_kind":"official","path":"nmodel.py","file_url":"https://github.com/badger-rl/stop/blob/HEAD/nmodel.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"69b48660ab8e75d9"}},{"code_sha256_prefix":"5562d039a93af838","entry":"symsqrt","repo":"badger-rl/stop","repo_kind":"official","path":"nmodel.py","file_url":"https://github.com/badger-rl/stop/blob/HEAD/nmodel.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5562d039a93af838"}},{"code_sha256_prefix":"d51cc2bc73bd04cb","entry":"tanh","repo":"badger-rl/stop","repo_kind":"official","path":"nmodel.py","file_url":"https://github.com/badger-rl/stop/blob/HEAD/nmodel.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d51cc2bc73bd04cb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}