{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/open-the-black-box-step-based-policy-updates","title":"Open the Black Box: Step-based Policy Updates for Temporally-Correlated Episodic Reinforcement Learning","arxiv_id":"2401.11437","date":"2024-01-21","proceeding":null,"authors":["Ge Li","Hongyi Zhou","Dominik Roth","Serge Thilges","Fabian Otto","Rudolf Lioutikov","Gerhard Neumann"],"abstract":"Current advancements in reinforcement learning (RL) have predominantly focused on learning step-based policies that generate actions for each perceived state. While these methods efficiently leverage step information from environmental interaction, they often ignore the temporal correlation between actions, resulting in inefficient exploration and unsmooth trajectories that are challenging to implement on real hardware. Episodic RL (ERL) seeks to overcome these challenges by exploring in parameters space that capture the correlation of actions. However, these approaches typically compromise data efficiency, as they treat trajectories as opaque \\emph{black boxes}. In this work, we introduce a novel ERL algorithm, Temporally-Correlated Episodic RL (TCE), which effectively utilizes step information in episodic policy updates, opening the 'black box' in existing ERL methods while retaining the smooth and consistent exploration in parameter space. TCE synergistically combines the advantages of step-based and episodic RL, achieving comparable performance to recent ERL methods while maintaining data efficiency akin to state-of-the-art (SoTA) step-based RL.","url_abs":"https://arxiv.org/abs/2401.11437v1","url_pdf":"https://arxiv.org/pdf/2401.11437v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"open-the-black-box-step-based-policy-updates","repo_url":"https://github.com/brucegeli/tce_rl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"GPL-3.0"}}],"tasks":[{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2401.11437","atlas_url":"https://app.syntology.ai/?focus=2401.11437","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.11437"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/brucegeli/tce_rl","reach":{"status":"ok","spdx":"GPL-3.0"}}],"summary":{"ran":7,"unverified":2},"by_repo_kind":{"official":{"samples":9,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"92e5211355eecee6","entry":"dir_go_up","repo":"brucegeli/tce_rl","repo_kind":"official","path":"mprl/util/util_file.py","file_url":"https://github.com/brucegeli/tce_rl/blob/HEAD/mprl/util/util_file.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"92e5211355eecee6"}},{"code_sha256_prefix":"2716f92c20b5f303","entry":"from_string_to_array","repo":"brucegeli/tce_rl","repo_kind":"official","path":"mprl/util/util_data_structure.py","file_url":"https://github.com/brucegeli/tce_rl/blob/HEAD/mprl/util/util_data_structure.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"2716f92c20b5f303"}},{"code_sha256_prefix":"791d15c2f8feff67","entry":"get_dataset_dir","repo":"brucegeli/tce_rl","repo_kind":"official","path":"mprl/util/util_file.py","file_url":"https://github.com/brucegeli/tce_rl/blob/HEAD/mprl/util/util_file.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"791d15c2f8feff67"}},{"code_sha256_prefix":"20f320988be5d87f","entry":"git_repos_old_vs_new","repo":"brucegeli/tce_rl","repo_kind":"official","path":"mprl/util/util_git.py","file_url":"https://github.com/brucegeli/tce_rl/blob/HEAD/mprl/util/util_git.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"20f320988be5d87f"}},{"code_sha256_prefix":"cdb63a3d0d6500be","entry":"make_iterable","repo":"brucegeli/tce_rl","repo_kind":"official","path":"mprl/util/util_data_structure.py","file_url":"https://github.com/brucegeli/tce_rl/blob/HEAD/mprl/util/util_data_structure.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"cdb63a3d0d6500be"}},{"code_sha256_prefix":"d628a55b9fcdd727","entry":"parse_dtype_device","repo":"brucegeli/tce_rl","repo_kind":"official","path":"mprl/util/util_data_structure.py","file_url":"https://github.com/brucegeli/tce_rl/blob/HEAD/mprl/util/util_data_structure.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"d628a55b9fcdd727"}},{"code_sha256_prefix":"2989d2eb82d281fe","entry":"remove_file_dir","repo":"brucegeli/tce_rl","repo_kind":"official","path":"mprl/util/util_file.py","file_url":"https://github.com/brucegeli/tce_rl/blob/HEAD/mprl/util/util_file.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"2989d2eb82d281fe"}},{"code_sha256_prefix":"9c8d0df3d467a118","entry":"how_fast","repo":"brucegeli/tce_rl","repo_kind":"official","path":"mprl/util/util_debug.py","file_url":"https://github.com/brucegeli/tce_rl/blob/HEAD/mprl/util/util_debug.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"9c8d0df3d467a118"}},{"code_sha256_prefix":"bac2eb5a8cec567f","entry":"run_time_test","repo":"brucegeli/tce_rl","repo_kind":"official","path":"mprl/util/util_debug.py","file_url":"https://github.com/brucegeli/tce_rl/blob/HEAD/mprl/util/util_debug.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"GPL-3.0","inline_ok":false,"mcp_get_code":{"code_sha256":"bac2eb5a8cec567f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}