{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/time-discretization-invariant-safe-action","title":"Time Discretization-Invariant Safe Action Repetition for Policy Gradient Methods","arxiv_id":"2111.03941","date":"2021-11-06","proceeding":"NeurIPS 2021 12","authors":["Seohong Park","Jaekyeom Kim","Gunhee Kim"],"abstract":"In reinforcement learning, continuous time is often discretized by a time scale $\\delta$, to which the resulting performance is known to be highly sensitive. In this work, we seek to find a $\\delta$-invariant algorithm for policy gradient (PG) methods, which performs well regardless of the value of $\\delta$. We first identify the underlying reasons that cause PG methods to fail as $\\delta \\to 0$, proving that the variance of the PG estimator can diverge to infinity in stochastic environments under a certain assumption of stochasticity. While durative actions or action repetition can be employed to have $\\delta$-invariance, previous action repetition methods cannot immediately react to unexpected situations in stochastic environments. We thus propose a novel $\\delta$-invariant method named Safe Action Repetition (SAR) applicable to any existing PG algorithm. SAR can handle the stochasticity of environments by adaptively reacting to changes in states during action repetition. We empirically show that our method is not only $\\delta$-invariant but also robust to stochasticity, outperforming previous $\\delta$-invariant approaches on eight MuJoCo environments with both deterministic and stochastic settings. Our code is available at https://vision.snu.ac.kr/projects/sar.","url_abs":"https://arxiv.org/abs/2111.03941v6","url_pdf":"https://arxiv.org/pdf/2111.03941v6.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"time-discretization-invariant-safe-action","repo_url":"https://github.com/artberryx/SAR","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"mujoco","task_name":"MuJoCo"},{"task_slug":"policy-gradient-methods","task_name":"Policy Gradient Methods"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2111.03941","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2111.03941"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/artberryx/SAR","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":16},"by_repo_kind":{"official":{"samples":16,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5260262d19bd2712","entry":"conv","repo":"artberryx/SAR","repo_kind":"official","path":"sb/stable_baselines/common/tf_layers.py","file_url":"https://github.com/artberryx/SAR/blob/HEAD/sb/stable_baselines/common/tf_layers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5260262d19bd2712"}},{"code_sha256_prefix":"ef103f787c48beca","entry":"create_mlp","repo":"artberryx/SAR","repo_kind":"official","path":"sb3/stable_baselines3/common/torch_layers.py","file_url":"https://github.com/artberryx/SAR/blob/HEAD/sb3/stable_baselines3/common/torch_layers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ef103f787c48beca"}},{"code_sha256_prefix":"aad776721400d1cc","entry":"get_action_dim","repo":"artberryx/SAR","repo_kind":"official","path":"sb/repeat/utils.py","file_url":"https://github.com/artberryx/SAR/blob/HEAD/sb/repeat/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aad776721400d1cc"}},{"code_sha256_prefix":"f2ded9c22d608f44","entry":"get_actor_critic_arch","repo":"artberryx/SAR","repo_kind":"official","path":"sb3/stable_baselines3/common/torch_layers.py","file_url":"https://github.com/artberryx/SAR/blob/HEAD/sb3/stable_baselines3/common/torch_layers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2ded9c22d608f44"}},{"code_sha256_prefix":"0d0ddaf36545261e","entry":"get_monitor_files","repo":"artberryx/SAR","repo_kind":"official","path":"sb/stable_baselines/bench/monitor.py","file_url":"https://github.com/artberryx/SAR/blob/HEAD/sb/stable_baselines/bench/monitor.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d0ddaf36545261e"}},{"code_sha256_prefix":"dbe46349368917f5","entry":"load_results","repo":"artberryx/SAR","repo_kind":"official","path":"sb/stable_baselines/bench/monitor.py","file_url":"https://github.com/artberryx/SAR/blob/HEAD/sb/stable_baselines/bench/monitor.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dbe46349368917f5"}},{"code_sha256_prefix":"42dca35d1e009c9a","entry":"make_atari","repo":"artberryx/SAR","repo_kind":"official","path":"sb/stable_baselines/common/atari_wrappers.py","file_url":"https://github.com/artberryx/SAR/blob/HEAD/sb/stable_baselines/common/atari_wrappers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"42dca35d1e009c9a"}},{"code_sha256_prefix":"d2ac8a40c8413065","entry":"make_output_format","repo":"artberryx/SAR","repo_kind":"official","path":"sb/stable_baselines/logger.py","file_url":"https://github.com/artberryx/SAR/blob/HEAD/sb/stable_baselines/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d2ac8a40c8413065"}},{"code_sha256_prefix":"b1b0b13946e3393b","entry":"mlp","repo":"artberryx/SAR","repo_kind":"official","path":"sb/stable_baselines/common/tf_layers.py","file_url":"https://github.com/artberryx/SAR/blob/HEAD/sb/stable_baselines/common/tf_layers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1b0b13946e3393b"}},{"code_sha256_prefix":"90935b4a6d887717","entry":"ortho_init","repo":"artberryx/SAR","repo_kind":"official","path":"sb/stable_baselines/common/tf_layers.py","file_url":"https://github.com/artberryx/SAR/blob/HEAD/sb/stable_baselines/common/tf_layers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"90935b4a6d887717"}},{"code_sha256_prefix":"a3fd9c3a978a7624","entry":"rolling_window","repo":"artberryx/SAR","repo_kind":"official","path":"sb/stable_baselines/results_plotter.py","file_url":"https://github.com/artberryx/SAR/blob/HEAD/sb/stable_baselines/results_plotter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a3fd9c3a978a7624"}},{"code_sha256_prefix":"75fb406b970614d4","entry":"summary_val","repo":"artberryx/SAR","repo_kind":"official","path":"sb/stable_baselines/logger.py","file_url":"https://github.com/artberryx/SAR/blob/HEAD/sb/stable_baselines/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"75fb406b970614d4"}},{"code_sha256_prefix":"9cc2d7b8e6215c80","entry":"ts2xy","repo":"artberryx/SAR","repo_kind":"official","path":"sb/stable_baselines/results_plotter.py","file_url":"https://github.com/artberryx/SAR/blob/HEAD/sb/stable_baselines/results_plotter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9cc2d7b8e6215c80"}},{"code_sha256_prefix":"dbdf57c6ccdd4893","entry":"valid_float_value","repo":"artberryx/SAR","repo_kind":"official","path":"sb/stable_baselines/logger.py","file_url":"https://github.com/artberryx/SAR/blob/HEAD/sb/stable_baselines/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dbdf57c6ccdd4893"}},{"code_sha256_prefix":"0680d6189aa0a2ff","entry":"window_func","repo":"artberryx/SAR","repo_kind":"official","path":"sb/stable_baselines/results_plotter.py","file_url":"https://github.com/artberryx/SAR/blob/HEAD/sb/stable_baselines/results_plotter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0680d6189aa0a2ff"}},{"code_sha256_prefix":"3833ca08c133f805","entry":"wrap_deepmind","repo":"artberryx/SAR","repo_kind":"official","path":"sb/stable_baselines/common/atari_wrappers.py","file_url":"https://github.com/artberryx/SAR/blob/HEAD/sb/stable_baselines/common/atari_wrappers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3833ca08c133f805"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}