{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/resetting-the-optimizer-in-deep-rl-an","title":"Resetting the Optimizer in Deep RL: An Empirical Study","arxiv_id":"2306.17833","date":"2023-06-30","proceeding":"NeurIPS 2023 11","authors":[],"abstract":"We focus on the task of approximating the optimal value function in deep reinforcement learning. This iterative process is comprised of solving a sequence of optimization problems where the loss function changes per iteration. The common approach to solving this sequence of problems is to employ modern variants of the stochastic gradient descent algorithm such as Adam. These optimizers maintain their own internal parameters such as estimates of the first-order and the second-order moments of the gradient, and update them over time. Therefore, information obtained in previous iterations is used to solve the optimization problem in the current iteration. We demonstrate that this can contaminate the moment estimates because the optimization landscape can change arbitrarily from one iteration to the next one. To hedge against this negative effect, a simple idea is to reset the internal parameters of the optimizer when starting a new iteration. We empirically investigate this resetting idea by employing various optimizers in conjunction with the Rainbow algorithm. We demonstrate that this simple modification significantly improves the performance of deep RL on the Atari benchmark.","url_abs":"https://arxiv.org/abs/2306.17833v2","url_pdf":"https://arxiv.org/pdf/2306.17833v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.17833","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.17833"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Kaixhin/Rainbow","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/devsisters/DQN-tensorflow","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"found_in_text":{"samples":7,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3a0e13b504c8be57","entry":"class_vars","repo":"devsisters/DQN-tensorflow","repo_kind":"found_in_text","path":"dqn/base.py","file_url":"https://github.com/devsisters/DQN-tensorflow/blob/HEAD/dqn/base.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3a0e13b504c8be57"}},{"code_sha256_prefix":"4cb9e720620b6eee","entry":"clipped_error","repo":"devsisters/DQN-tensorflow","repo_kind":"found_in_text","path":"dqn/ops.py","file_url":"https://github.com/devsisters/DQN-tensorflow/blob/HEAD/dqn/ops.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4cb9e720620b6eee"}},{"code_sha256_prefix":"d2447e0a77e70092","entry":"get_config","repo":"devsisters/DQN-tensorflow","repo_kind":"found_in_text","path":"config.py","file_url":"https://github.com/devsisters/DQN-tensorflow/blob/HEAD/config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d2447e0a77e70092"}},{"code_sha256_prefix":"5622fc4882566cdc","entry":"linear","repo":"devsisters/DQN-tensorflow","repo_kind":"found_in_text","path":"dqn/ops.py","file_url":"https://github.com/devsisters/DQN-tensorflow/blob/HEAD/dqn/ops.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5622fc4882566cdc"}},{"code_sha256_prefix":"d49a89bcb28274d4","entry":"load_pkl","repo":"devsisters/DQN-tensorflow","repo_kind":"found_in_text","path":"dqn/utils.py","file_url":"https://github.com/devsisters/DQN-tensorflow/blob/HEAD/dqn/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d49a89bcb28274d4"}},{"code_sha256_prefix":"6f76cce2b6054e5f","entry":"rgb2gray","repo":"devsisters/DQN-tensorflow","repo_kind":"found_in_text","path":"dqn/utils.py","file_url":"https://github.com/devsisters/DQN-tensorflow/blob/HEAD/dqn/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6f76cce2b6054e5f"}},{"code_sha256_prefix":"db4c70416d7901da","entry":"timeit","repo":"devsisters/DQN-tensorflow","repo_kind":"found_in_text","path":"dqn/utils.py","file_url":"https://github.com/devsisters/DQN-tensorflow/blob/HEAD/dqn/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"db4c70416d7901da"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}