{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/acting-in-delayed-environments-with-non-1","title":"Acting in Delayed Environments with Non-Stationary Markov Policies","arxiv_id":"2101.11992","date":"2021-01-28","proceeding":"ICLR 2021 1","authors":["Esther Derman","Gal Dalal","Shie Mannor"],"abstract":"The standard Markov Decision Process (MDP) formulation hinges on the assumption that an action is executed immediately after it was chosen. However, assuming it is often unrealistic and can lead to catastrophic failures in applications such as robotic manipulation, cloud computing, and finance. We introduce a framework for learning and planning in MDPs where the decision-maker commits actions that are executed with a delay of $m$ steps. The brute-force state augmentation baseline where the state is concatenated to the last $m$ committed actions suffers from an exponential complexity in $m$, as we show for policy iteration. We then prove that with execution delay, deterministic Markov policies in the original state-space are sufficient for attaining maximal reward, but need to be non-stationary. As for stationary Markov policies, we show they are sub-optimal in general. Consequently, we devise a non-stationary Q-learning style model-based algorithm that solves delayed execution tasks without resorting to state-augmentation. Experiments on tabular, physical, and Atari domains reveal that it converges quickly to high performance even for substantial delays, while standard approaches that either ignore the delay or rely on state-augmentation struggle or fail due to divergence. The code is available at github.com/galdl/rl_delay_basic and github.com/galdl/rl_delay_atari.","url_abs":"https://arxiv.org/abs/2101.11992v4","url_pdf":"https://arxiv.org/pdf/2101.11992v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"acting-in-delayed-environments-with-non-1","repo_url":"https://github.com/galdl/rl_delay_atari","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"acting-in-delayed-environments-with-non-1","repo_url":"https://github.com/galdl/rl_delay_basic","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"cloud-computing","task_name":"Cloud Computing"},{"task_slug":"q-learning","task_name":"Q-Learning"}],"methods":[{"method_slug":"q-learning","method_name":"Q-Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2101.11992","atlas_url":"https://app.syntology.ai/?focus=2101.11992","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2101.11992"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/galdl/rl_delay_basic","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/galdl/rl_delay_atari","reach":{"status":"unanswered"}}],"summary":{"ran":1,"ran_fixture":2,"unverified":4},"by_repo_kind":{"official":{"samples":7,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bac53a374bb21edc","entry":"ForwardModel","repo":"galdl/rl_delay_basic","repo_kind":"official","path":"dqn_agents.py","file_url":"https://github.com/galdl/rl_delay_basic/blob/HEAD/dqn_agents.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bac53a374bb21edc"}},{"code_sha256_prefix":"64ee35f5621332c6","entry":"concatenate_state_action","repo":"galdl/rl_delay_basic","repo_kind":"official","path":"dqn_agents.py","file_url":"https://github.com/galdl/rl_delay_basic/blob/HEAD/dqn_agents.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"64ee35f5621332c6"}},{"code_sha256_prefix":"b4c93343c916d898","entry":"reshape_state","repo":"galdl/rl_delay_basic","repo_kind":"official","path":"dqn_agents.py","file_url":"https://github.com/galdl/rl_delay_basic/blob/HEAD/dqn_agents.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b4c93343c916d898"}},{"code_sha256_prefix":"5b1a8d1d169f146e","entry":"DDQNAgent","repo":"galdl/rl_delay_basic","repo_kind":"official","path":"dqn_agents.py","file_url":"https://github.com/galdl/rl_delay_basic/blob/HEAD/dqn_agents.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5b1a8d1d169f146e"}},{"code_sha256_prefix":"c0594047adeb6fb6","entry":"DDQNPlanningAgent","repo":"galdl/rl_delay_basic","repo_kind":"official","path":"dqn_agents.py","file_url":"https://github.com/galdl/rl_delay_basic/blob/HEAD/dqn_agents.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c0594047adeb6fb6"}},{"code_sha256_prefix":"96778dedd8f50887","entry":"DQNAgent","repo":"galdl/rl_delay_basic","repo_kind":"official","path":"dqn_agents.py","file_url":"https://github.com/galdl/rl_delay_basic/blob/HEAD/dqn_agents.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"96778dedd8f50887"}},{"code_sha256_prefix":"3880ba2e2c852b2c","entry":"update_loss","repo":"galdl/rl_delay_basic","repo_kind":"official","path":"dqn_agents.py","file_url":"https://github.com/galdl/rl_delay_basic/blob/HEAD/dqn_agents.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3880ba2e2c852b2c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}