{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/efficient-deep-reinforcement-learning","title":"Efficient Deep Reinforcement Learning Requires Regulating Overfitting","arxiv_id":"2304.10466","date":"2023-04-20","proceeding":null,"authors":["Qiyang Li","Aviral Kumar","Ilya Kostrikov","Sergey Levine"],"abstract":"Deep reinforcement learning algorithms that learn policies by trial-and-error must learn from limited amounts of data collected by actively interacting with the environment. While many prior works have shown that proper regularization techniques are crucial for enabling data-efficient RL, a general understanding of the bottlenecks in data-efficient RL has remained unclear. Consequently, it has been difficult to devise a universal technique that works well across all domains. In this paper, we attempt to understand the primary bottleneck in sample-efficient deep RL by examining several potential hypotheses such as non-stationarity, excessive action distribution shift, and overfitting. We perform thorough empirical analysis on state-based DeepMind control suite (DMC) tasks in a controlled and systematic way to show that high temporal-difference (TD) error on the validation set of transitions is the main culprit that severely affects the performance of deep RL algorithms, and prior methods that lead to good performance do in fact, control the validation TD error to be low. This observation gives us a robust principle for making deep RL efficient: we can hill-climb on the validation TD error by utilizing any form of regularization techniques from supervised learning. We show that a simple online model selection method that targets the validation TD error is effective across state-based DMC and Gym tasks.","url_abs":"https://arxiv.org/abs/2304.10466v1","url_pdf":"https://arxiv.org/pdf/2304.10466v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"deep-reinforcement-learning","task_name":"Deep Reinforcement Learning"},{"task_slug":"model-selection","task_name":"Model Selection"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2304.10466","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2304.10466"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ikostrikov/walk_in_the_park","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/ikostrikov/jaxrl","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/evgenii-nikishin/rl_with_resets","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":14},"by_repo_kind":{"found_in_text":{"samples":15,"ran":1,"repositories":3}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"718462716d78ea99","entry":"default_init","repo":"evgenii-nikishin/rl_with_resets","repo_kind":"found_in_text","path":"continuous_control/networks/common.py","file_url":"https://github.com/evgenii-nikishin/rl_with_resets/blob/HEAD/continuous_control/networks/common.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"718462716d78ea99"}},{"code_sha256_prefix":"f0ff6ea848e52f45","entry":"REDQLearner","repo":"ikostrikov/jaxrl","repo_kind":"found_in_text","path":"jaxrl/agents/redq/redq_learner.py","file_url":"https://github.com/ikostrikov/jaxrl/blob/HEAD/jaxrl/agents/redq/redq_learner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f0ff6ea848e52f45"}},{"code_sha256_prefix":"cde7d367f320d964","entry":"_update_jit","repo":"ikostrikov/jaxrl","repo_kind":"found_in_text","path":"jaxrl/agents/redq/redq_learner.py","file_url":"https://github.com/ikostrikov/jaxrl/blob/HEAD/jaxrl/agents/redq/redq_learner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cde7d367f320d964"}},{"code_sha256_prefix":"b0d2f7580e14b4a6","entry":"create_env_wrapper","repo":"evgenii-nikishin/rl_with_resets","repo_kind":"found_in_text","path":"discrete_control/eval_run_experiment.py","file_url":"https://github.com/evgenii-nikishin/rl_with_resets/blob/HEAD/discrete_control/eval_run_experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b0d2f7580e14b4a6"}},{"code_sha256_prefix":"a50b18c10731aa3a","entry":"delete_ind_from_array","repo":"evgenii-nikishin/rl_with_resets","repo_kind":"found_in_text","path":"discrete_control/eval_run_experiment.py","file_url":"https://github.com/evgenii-nikishin/rl_with_resets/blob/HEAD/discrete_control/eval_run_experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a50b18c10731aa3a"}},{"code_sha256_prefix":"58031134f80381f9","entry":"drq_image_aug","repo":"evgenii-nikishin/rl_with_resets","repo_kind":"found_in_text","path":"discrete_control/networks.py","file_url":"https://github.com/evgenii-nikishin/rl_with_resets/blob/HEAD/discrete_control/networks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"58031134f80381f9"}},{"code_sha256_prefix":"a34f6a7ad7ae7710","entry":"evaluate","repo":"evgenii-nikishin/rl_with_resets","repo_kind":"found_in_text","path":"continuous_control/evaluation.py","file_url":"https://github.com/evgenii-nikishin/rl_with_resets/blob/HEAD/continuous_control/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a34f6a7ad7ae7710"}},{"code_sha256_prefix":"fed35e39e159f251","entry":"evaluate","repo":"ikostrikov/walk_in_the_park","repo_kind":"found_in_text","path":"rl/evaluation.py","file_url":"https://github.com/ikostrikov/walk_in_the_park/blob/HEAD/rl/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fed35e39e159f251"}},{"code_sha256_prefix":"71d1b50edc65f329","entry":"normalize_score","repo":"evgenii-nikishin/rl_with_resets","repo_kind":"found_in_text","path":"discrete_control/eval_run_experiment.py","file_url":"https://github.com/evgenii-nikishin/rl_with_resets/blob/HEAD/discrete_control/eval_run_experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"71d1b50edc65f329"}},{"code_sha256_prefix":"5ef98ddf8676a4f9","entry":"orientation_terminal_condition","repo":"ikostrikov/walk_in_the_park","repo_kind":"found_in_text","path":"real/resetters.py","file_url":"https://github.com/ikostrikov/walk_in_the_park/blob/HEAD/real/resetters.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5ef98ddf8676a4f9"}},{"code_sha256_prefix":"055c05453cdea31d","entry":"process_inputs","repo":"evgenii-nikishin/rl_with_resets","repo_kind":"found_in_text","path":"discrete_control/networks.py","file_url":"https://github.com/evgenii-nikishin/rl_with_resets/blob/HEAD/discrete_control/networks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"055c05453cdea31d"}},{"code_sha256_prefix":"3692fa3a61b257cd","entry":"renormalize","repo":"evgenii-nikishin/rl_with_resets","repo_kind":"found_in_text","path":"discrete_control/networks.py","file_url":"https://github.com/evgenii-nikishin/rl_with_resets/blob/HEAD/discrete_control/networks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3692fa3a61b257cd"}},{"code_sha256_prefix":"390aef8497c3346d","entry":"soft_target_update","repo":"ikostrikov/walk_in_the_park","repo_kind":"found_in_text","path":"rl/networks/common.py","file_url":"https://github.com/ikostrikov/walk_in_the_park/blob/HEAD/rl/networks/common.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"390aef8497c3346d"}},{"code_sha256_prefix":"99bd73397c2663ca","entry":"split_tree","repo":"evgenii-nikishin/rl_with_resets","repo_kind":"found_in_text","path":"continuous_control/networks/common.py","file_url":"https://github.com/evgenii-nikishin/rl_with_resets/blob/HEAD/continuous_control/networks/common.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"99bd73397c2663ca"}},{"code_sha256_prefix":"9678fbfc085148cf","entry":"tree_norm","repo":"evgenii-nikishin/rl_with_resets","repo_kind":"found_in_text","path":"continuous_control/networks/common.py","file_url":"https://github.com/evgenii-nikishin/rl_with_resets/blob/HEAD/continuous_control/networks/common.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9678fbfc085148cf"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}