{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improving-deep-reinforcement-learning-by","title":"Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn","arxiv_id":"2409.04792","date":"2024-09-07","proceeding":null,"authors":["Hongyao Tang","Glen Berseth"],"abstract":"Deep neural networks provide Reinforcement Learning (RL) powerful function approximators to address large-scale decision-making problems. However, these approximators introduce challenges due to the non-stationary nature of RL training. One source of the challenges in RL is that output predictions can churn, leading to uncontrolled changes after each batch update for states not included in the batch. Although such a churn phenomenon exists in each step of network training, how churn occurs and impacts RL remains under-explored. In this work, we start by characterizing churn in a view of Generalized Policy Iteration with function approximation, and we discover a chain effect of churn that leads to a cycle where the churns in value estimation and policy improvement compound and bias the learning dynamics throughout the iteration. Further, we concretize the study and focus on the learning issues caused by the chain effect in different settings, including greedy action deviation in value-based methods, trust region violation in proximal policy optimization, and dual bias of policy value in actor-critic methods. We then propose a method to reduce the chain effect across different settings, called Churn Approximated ReductIoN (CHAIN), which can be easily plugged into most existing DRL algorithms. Our experiments demonstrate the effectiveness of our method in both reducing churn and improving learning performance across online and offline, value-based and policy-based RL settings, as well as a scaling setting.","url_abs":"https://arxiv.org/abs/2409.04792v2","url_pdf":"https://arxiv.org/pdf/2409.04792v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improving-deep-reinforcement-learning-by","repo_url":"https://github.com/bluecontra/CHAIN","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.04792","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.04792"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bluecontra/CHAIN","reach":null}],"summary":{"ran":3,"ran_honours":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"6f2dd65ce04506ad","entry":"DoubleDQN_CHAIN","repo":"bluecontra/CHAIN","repo_kind":"official","path":"minatar/agents/double_dqn_chain.py","file_url":"https://github.com/bluecontra/CHAIN/blob/HEAD/minatar/agents/double_dqn_chain.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6f2dd65ce04506ad"}},{"code_sha256_prefix":"2bc9d20a97ad4b7c","entry":"QNetwork","repo":"bluecontra/CHAIN","repo_kind":"official","path":"minatar/agents/double_dqn_chain.py","file_url":"https://github.com/bluecontra/CHAIN/blob/HEAD/minatar/agents/double_dqn_chain.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2bc9d20a97ad4b7c"}},{"code_sha256_prefix":"519bcb97d620e3bb","entry":"evaluate_policy","repo":"bluecontra/chain","repo_kind":"official","path":"minatar/run_double_dqn_chain.py","file_url":"https://github.com/bluecontra/chain/blob/HEAD/minatar/run_double_dqn_chain.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"519bcb97d620e3bb"}},{"code_sha256_prefix":"878357e874462801","entry":"replay_buffer","repo":"bluecontra/CHAIN","repo_kind":"official","path":"minatar/agents/double_dqn_chain.py","file_url":"https://github.com/bluecontra/CHAIN/blob/HEAD/minatar/agents/double_dqn_chain.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"878357e874462801"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}